Large Data Sets

Example code from "Handling Large Data Sets Efficiently in MATLAB " webinar (http://www.mathworks.com/company/events/we.../wbnr30435.html) describing strategies for handling large amounts of data in MATLAB and avoiding "out-of-memory" errors. It will provide you with an understanding of the...

This webinar focuses on strategies and techniques for handling large amounts of data in MATLAB, including:d-deD? Understanding memory and its constraintsd-deD? Maximizing system resourcesd-deD? Minimizing your memory footprint in MATLABd-deD? Accessing memory across multiple systemsLeveraging...

TPIE is a software environment (written in C++) that facilitates the implementation of external memory algorithms. The goal of theoretical work in the area of external memory algorithms (also called I/O algorithms or out-of-core algorithms) has been to develop algorithms that minimize the...


Platforms: *nix

A MATLAB spectral clustering package to handle large data sets (200,000 RCV1 data) on a 4GB memory general machine. We implement various ways of approximating the dense similarity matrix, including nearest neighbors and the Nystrom method.


Platforms: Windows, Mac, BSD

RNEL_DB is an object oriented database solution for large data sets that need to be stored and/or analyzed within Matlab. It provides a transparent solution for people that currently store their data in matlab-structures.Using the RNEL_DB class-definition, we created a database solution that:* is...


Platforms: Matlab

Statistics::Descriptive::Discrete is a Perl module to compute descriptive statistics for discrete data sets. SYNOPSIS use Statistics::Descriptive::Discrete; my $stats = new Statistics::Descriptive::Discrete; $stats->add_data(1,10,2,0,1,4,5,1,10,8,7); print "count = ",$stats->count(),"n";...


Platforms: *nix

MPCA is a comprehensive suite of tools for doing discrete principal components analysis on data sets of size 100Mb or more. Scaling is done using sparse vectors, multi-threading, memory mapping, and other POSIX tricks. Reports, file dumping utilities and other utilities are included. The...


Platforms: *nix

I wrote this to plot some overlapping data sets with there means and regions of +/- std deviations shown. It could be used for other purposes though.


Platforms: Matlab

IVuPy (I-View-Py) aims to be a solid basis for large Qt based Python programs geared to data analysis and 3D visualization of huge data sets. Python is extended by IVuPy with more than 600 classes of two of the Coin3D C++ class libraries: Coin and SoQt. Data exchange between Python and the...


Platforms: *nix

World Wind 3D Data Viewer is a program developed using NASA's World Wind Java Software Development Kit (SDK) that will enable you to view Australia's continental data sets. The viewer allows you to compare national data sets such as the radioelements, the gravity and magnetic anomalies, and...


Platforms: Windows

Comma-Separated Values (CSV) is a widespread, cross-platform file format for exchanging structured data sets. For example, Microsoft Office Outlook imports contacts stored in CSV format (called "DAT data files"). Microsoft Office Excel offers bi-directional (read and write) support for CSV files....


Platforms: Windows

Modular toolkit for Data Processing (MDP) is a Python data processing framework. Implemented algorithms include: Principal Component Analysis (PCA), Independent Component Analysis (ICA), Slow Feature Analysis (SFA), Growing Neural Gas (GNG), Factor Analysis, Fisher Discriminant Analysis (FDA),...


Platforms: Windows, Mac, *nix, Python, BSD Solaris

Set EXIF Data sets, not edits, the EXIF data in an image. Set EXIF Data uses the excellent EXIFtool (comes bundled), a tool which you should install first. What's New in This Release: [ read full changelog ] New: ?*A* Added the ability to add or subtract a fixed time to/from a batch of photo's....


Platforms: Mac

Slim is a data compression system for scientific data sets, a binary and a library with C linkage. Slim works with integer data from one or more channels in a file, which it can compress more effectively and more rapidly than general tools like gzip.


Platforms: *nix

This module tries to find middle ground between one at a time and all at once processing of data sets. The purpose of this module is to avoid the overhead of implementing an iterative api when this isn't necessary, without breaking forward compatibility in case that becomes necessary later on....


Platforms: *nix

data-gc-ca-api: a simple python api for the Canada Open Data Portal The Government of Canada recently released a number of open data sets at the website www.data.gc.ca. This simple python package has tools for accessing the City Weather open data set. It could be expanded to include more.


Platforms: *nix

Transparent Parallel I/O Environment is a software environment (written in C++) that facilitates the implementation of external memory algorithms. The goal of theoretical work in the area of external memory algorithms (also called I/O algorithms or out-of-core algorithms) has been to develop...


Platforms: *nix

GMT is an open source collection of ~60 tools for manipulating geographic and Cartesian data sets (including filtering, trend fitting, gridding, projecting, etc.) and producing Encapsulated PostScript File (EPS) illustrations ranging from simple x-y plots via contour maps to artificially...


Platforms: *nix

The Integrated Genome Browser (IGB, pronounced Ig-Bee) is an interactive, zoomable, scrollable software program you can use to visualize and explore genome-scale data sets, such as tiling array data, next-generation sequencing results, genome annotations, microarray designs, and the sequence...


Platforms: Windows

Scaffold Hunter is an accessible Java-based application built for the analysis of structure-related biochemical data. The program is able to facilitate the interactive exploration of chemical space by enabling generation of and navigation in a scaffold tree hierarchy annotated with various...


Platforms: Windows, Windows Vista, 7