Download Shareware and Freeware Software for Windows, Linux, Macintosh, PDA

line Home  |  About Us  |  Link To Us  |  FAQ  |  Contact

Serving Software Downloads in 976 Categories, Downloaded 32.389.664 Times

Sendtools 0.1.1

Company: Bryan Cole
Date Added: June 20, 2013  |  Visits: 243

Sendtools

Report Broken Link
Printer Friendly Version


Product Homepage
Download (16 downloads)

Sendtools is a collections of classes for efficiently consuming iterators into one or more data structures.<br /><br />Sendtools compliments the itertools module and other the excellent facilities Python offers for iteration. Sendtools is useful when:<br /><br /> * Your source iterator is too big to fit in memory<br /> * Your data source is I/O bounds so you don't want to make more than one pass<br /> * You want to collect data into two or more lists (or other collection)<br /> * You want to group, filter, transform or otherwise aggregate the data<br /><br />Such situations occur when you're analysing query-sets from large databases or datafiles (HDF5-files, for example).<br /><br />Sendtools is written using Cython to produce a 100% compiled module, for maximum performance.<br /><br />Requirements<br /><br />There are no dependencies outside of python to compile and install sendtools (although you will need a compiler obviously).<br /><br />If you want to hack on the Cython code, you'll need Cython-0.12.1 or later.<br />Installation<br /><br />Sendtools is installed from source using distutils in the usual way - run:<br /><br />python setup.py install<br /><br />to install it site-wide<br /><br />If you have Cython installed, you can also import the sendtools.pyx file directly using the pyximport module (part of Cython). This is handy for development, as used in the unittest script.<br /><br />Usage<br /><br />Sendtools is built on the concept of "Consumer" objects. These were inspired by python's generators (an early version of sendtools was implemented in python using generators). Consumer objects can have data "sent" into them. Unlike generators, Consumers do not produce data iteratively (no 'next' method), but they do produce a result which can be accessed at any time using the .result() method.<br /><br />Data is typically sent into a Consumer using the sendtools.send() function, which takes the form:<br /><br />output = send(source, target)<br /><br />where source is an iterator producing data. target is a Consumer object into which the data is sent. output is the Consumer's result, returned after the source has been fully consumed, or the Consumer indicates it's complete (by raising StopIteration), which ever happens first. Basically, the send function if a shortcut for writing a for-loop.<br /><br />The target may be list or set, representing the data structure you want to collect the data into. These are implicitly converted to Consumer objects by the send function. The input list (or set) is returned by the send function having been filled with data.<br /><br />Target can also be a (multiply nested) tuple of consumers. In this case the result will be a tuple which matches the structure of the target tuple, containing the results for each consumer. In this way, data from a source iterator can be collected into multiple lists in a single iteration pass.<br /><br />Sendtools defines many aggregation consumers. These do not produce a list or other collection as their result, but a scalar value.<br /><br />Examples<br /><br />Let's start with basic usage of the send() function:<br /><br />>>> from sendtools import send<br />>>> data = range(10)<br />>>> out=[]<br />>>> result = send(data, out)<br />>>> result<br />[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br />>>> out<br />[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br />>>> result is out<br />True<br /><br />The source 'data' is copied into the target 'out' and this is returned.<br /><br />Now lets see how to send data into multiple targets:<br /><br />>>> a, (b,c) = send(data, ([], ([],[])))<br />>>> a is b; b is c; a is c<br />False<br />False<br />False<br />>>> a == b; b == c; a == c<br />True<br />True<br />True<br /><br />The data is copied into three different lists.<br /><br />Data can be collected into sets as well as lists:<br /><br />>>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,"moose",4,2]<br />>>> send(data, set())<br />set([1, 2, 3, 4, 5, 6, 8, 'moose'])<br /><br />In fact, any MutableSequence or MutableSet (the Abstract Base Class) will do. Sadly, the std-lib array.array object is not registered as a MutableSequence out-the-box, but we can do this ourselves:<br /><br />>>> from array import array<br />>>> from collections import MutableSequence<br />>>> MutableSequence.register(array)<br />>>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,4,2]<br />>>> target = array("f") #an empty array<br />>>> send(data, target)<br />array('f', [1.0, 2.0, 3.0, 5.0, 4.0, 2.0, 6.0, 3.0, 4.0, 8.0, 5.0, 6.0, 3.0,<br />1.0, 5.0, 3.0, 6.0, 3.0, 6.0, 4.0, 2.0])<br /><br />Aggregation<br /><br />Now let's see some aggregation:<br /><br />>>> send(data, ([], (Max(), Min(), Sum(), Ave())))<br />([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], (9, 0, 45, 4.5))<br /><br />All the aggregation functions found in SQL are available: Sum, Max, Min, Ave, First, Last, Count.<br /><br />There are a few more besides these:<br /><br /> * Select - Picks the n'th item in a sequence<br /> * Stats - Computes an incremental standard deviation, mean and count of it's input.<br /><br />This last one only works with numerical input and returns a length-3 tuple as it's result.<br /><br />Transformations and Filtering<br /><br />Data can be filtered using Filter:<br /><br />>>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,4,2]<br />>>> send(data, Filter(lambda x:x%2==0, []))<br />[2, 4, 2, 6, 4, 8, 6, 6, 6, 4, 2]<br /><br />Data can be transformed using Map:<br /><br />>>> send(data, ([], Map(lambda x:x**2, [])))<br />([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 4, 9, 16, 25, 36, 49, 64, 81])<br /><br />One important use-case is splitting a sequence of tuples or other compound objects into multiple lists. Although this can be done with Map, this is such a common operation, we have a dedicated Get object for this purpose. eg.:<br /><br />>>> tups = [(x,x**2) for x in range(10)]<br />>>> print tups<br />[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25), (6, 36), (7, 49),<br />(8, 64), (9, 81)]<br />>>> a,b = send(tups, (Get(0,[]), Get(1,[])))<br />>>> a<br />[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]<br />>>> b<br />[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]<br /><br />This works for any suitable indexing object. For example, columns from a database query can be collected into some lists using this method. Object attributes can also be retrieved in a similar manner using the Attr object.<br /><br />Grouping and Switching<br /><br />Data can be grouped in a variety of ways. The grouping objects take a factory function as a keyword argument. This is called to create each group. By default, a list group is created, but more complex group-types are possible: aggregates, tuples of targets or even other grouping objects. Any valid target object can be used.<br /><br />Here's an example of simple grouping by number into sublists:<br /><br />>>> data<br />[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]<br />>>> send(data, GroupByN(3,[]))<br />[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 13, 14], [15, 16, 17]]<br /><br />Now let's use a more complex group factory for get the mean of each group, as well as the group list:<br /><br />>>> send(data, GroupByN(3, [], factory=lambda :([],Ave())))<br />[([0, 1, 2], 1.0), ([3, 4, 5], 4.0), ([6, 7, 8], 7.0), ([9, 10, 11], 10.0),<br />([12, 13, 14], 13.0), ([15, 16, 17], 16.0)]<br /><br />Groups can also be created using a key-function, with the GroupByKey object:<br /><br />>>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,4,2]<br />>>> send(data, GroupByKey(lambda x:x==5, []))<br />[[1, 2, 3], [5], [4, 2, 6, 3, 4, 8], [5], [6, 3, 1], [5], [3, 6, 3, 6, 4, 2]]<br /><br />Note, new groups are created whenever the key-function returns a different result to the previous item, regardless of whether that result has been used to create previous groups.<br /><br />Switching is a very close relative to grouping. The Switch object passes it's input to a key-function which must return an int. The input is passed to one of N outputs according to this int. I.e.<br /><br /> >>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,4,2]<br /> >>> send(data, Switch(lambda x:int(x>> data = [1,2,3,5,4,2,6,3,4,8,5,6,3,1,5,3,6,3,6,4,2]<br /> >>> func = lambda item: "low" if item>> send(data, SwitchByKey(func, init={"low":['foo']}))<br /> {'high': [5, 6, 8, 5, 6, 5, 6, 6],<br /> 'low': ['foo', 1, 2, 3, 4, 2, 3, 4, 3, 1, 3, 3, 4, 2]}<br /> >>> send(data, SwitchByKey(func, factory=Sum))<br /> {'high': 47, 'low': 35}<br /><br />The init keyword specifies a dictionary of groups with which to initialise the object (an empty dict by default). When a new key is encountered (that does not already exist in the dict), the factory function is called to create a new group for this key.<br /><br /><br />#md5=193c227fe8a14c39d891564cc47a0c87

Requirements: No special requirements
Platforms: *nix, Linux
Keyword: Consumer Cython Function Group Grouping Gtgtgt Import Input Lists Object Objects Produce Result Senddata Sendtools Source Target Tuple
Users rating: 0/10

License: Freeware Size: 10.24 KB
SENDTOOLS RELATED
Email  -  okEmail for Email System Utilities 1.1
okEmail is an ASP function to check proper email input, following these rules:-minimum 8 characters in total...-check for illegal characters-need an "@" in the email but only one is allowed-at least one "." must come after the "@"-must have at...
10 KB  
Email  -  okEmail 1.0
okEmail is an ASP function to check proper email input, following these rules: -minimum 8 characters in total... -check for illegal characters -need an "@" in the email but only one is allowed -at least one "." must come after the "@" -must have...
 
Miscellaneous  -  Claroline Course Exchange 1.1.1
The Claroline applet Claroline Course Exchange (CLCEX) allows Claroline users to import and export learning objects, for example learning paths or exercises, to and from learning object repositories (e.g. SWITCHcollection).
122.88 KB  
Business  -  FSuite CD 709
FreeSMUG - FreeSoftware Mac User Group, releases a collection CD with the most useful free and open source software applications packaged for Mac OS X. All 10.4 and PPC/Intel compatible. PPC and Intel Applications and Source CD are available. *...
688 MB  
Utilities  -  RangeCamera 1.0
You can measure the distance between you and the target very easily. You need only two simple action to do ! Input your height, and aim the camera to the target. This result is not exact as in a real measure. But when you want...
512 KB  
Development Tools  -  Robust function for only numeric values input 1.0
The function shows the text string S and prompt the user for an input value guaranteing that the input value is numdoTeric.When the value is not numdoTeric the function gives an error mensage and prompts for a new value until it is numeric.
10 KB  
Programming  -  entrypoint 0.1.5
A decorator to interact with argparse based on function signature. This is a decorator library that helps one to write small scripts in Python. There are three main features that it provides: * Automatically...
10.24 KB  
Programming  -  extensions for linux 0.4
extensions is a simple plugin system inspired from setuptools entry points. It allows an application to define and/or use plugins. How to define a plugin A plugin can be any callable object . It has to be registered through...
20.48 KB  
Libraries  -  Linux::Input::Joystick 1.02
Linux::Input::Joystick is a joystick-specific interface for Linux 2.2+ SYNOPSIS Usage use YAML; my $js = Linux::Input::Joystick->new(/dev/input/js0); while (1) { my @event = $js->poll(0.01); print Dump($_) foreach (@event); } This is...
6.14 KB  
Libraries  -  File::FilterFuncs 0.53
File::FilterFuncs is a Perl module that specify filter functions for files. SYNOPSIS use File::FilterFuncs qw(filters); filters(source.txt, sub { $_ = uc $_; 1 }, dest.txt ); INTRODUCTION File::FilterFuncs makes it easy to perform...
12.29 KB  
NEW DOWNLOADS IN LINUX SOFTWARE, UTILITIES
Linux Software  -  wpCache® WordPress HTTP Cache 1.9
wpCache® is a high-performance, distributed object, caching system application, generic in nature, but intended for use in speeding up dynamic web applications, by decreasing database load time. wpCache® decreases dramatically the page...
3.51 MB  
Linux Software  -  Polling Autodialer Software 3.4
ICTBroadcast Auto Dialer software has a survey campaign for telephone surveys and polls. This auto dialer software automatically dials a list of numbers and asks them a set of questions that they can respond to, by using their telephone keypad....
488 B  
Linux Software  -  Total Video Converter Mac Free 3.5.5
Total Video Converter Mac Free developed by EffectMatrix Ltd is the official legal version of Total Video Converter which was a globally recognized brand since 2006. Total Video Converter Mac Free is a free but powerful all-in-one video...
17.7 MB  
Linux Software  -  Skeith mod_log_sql Analyzer 2.10beta2
Skeith is a php based front end for analyzing logs for Apache using mod_log_sql.
47.5 KB  
Linux Software  -  SLAX 6.0+
Slax is a modern, portable, small and fast Linux operating system with a modular approach and outstanding design. Despite its small size, Slax provides a wide collection of pre-installed software for daily use, including a well organized graphical...
190 KB  
Utilities  -  LPAR2RRD 4.95-4
LPAR2RRD collects performance data and generates actual, historical and future trends utilization graphs of your virtual environment. It is agentless (it receives everything from the management stations like vCenter or HMC). The product supports...
2.25 MB  
Utilities  -  Nessconnect 1.0.2
Nessconnect is a GUI, CLI and API client for Nessus and Nessus compatible servers. With an improved user interface, it provides local session management, scan templates, report generation through XSLT, charts and graphs, and vulnerability trending.
819.2 KB  
Utilities  -  Dynamic Power Management 2.6.16
The Dynamic Power Management (DPM) project explores technologies to improve power conservation capabilities of platforms based on open source software. Of particular interest are techniques applicable to running systems, adjusting power parameters...
30.72 KB  
Utilities  -  Ethernet bridge tables 2.4.37.9
Ethernet bridge tables - Linux Ethernet filter for the Linux bridge. The 2.4-ebtables-brnf package contains the ebtables+bridge-nf patch. Be sure to check out the ebtables hp. This site also contains the arptables userspace tool.
40.96 KB  
Utilities  -  SaraB 1.0.0
SaraB works with DAR (Disk ARchive) to schedule and rotate backups on random-access media (i.e. hard drives, CDs, DVDs, Zip, etc. Basically anything except magnetic tapes.) This reduces hassle for the administrator by providing an automatic backup...
20.48 KB