Tag Archives: Dev

Iterables vs. Generators in Python

Iterables vs. Generators in Python

I’ve had some people asking me lately what the difference between a Python iteratable and a Python generator is, and when to use them. This is a short write-up to show some of the differences and the benefits of use.

Iterables

Simply put, an iteratable in Python is any object that will allow you to use it in with an in operator, e.g.:

    A = [1,2,3,4,5]
    for a in A:
        print(a)

will output:

1
2
3
4
5

An iteratable is an object that will hold other items, primitives or more complex objects. Lists, tuples, sets, dicts are all examples of iterables. So are things like strings and files. We use them all the time, quite properly, in our code.

And when we use a comprehension, (hint, hint!) we are also creating an iterable, so

    B = [x for x in range(1,10000)]

creates a list of 100 values, from 0…10000.

So that’s great. but perhaps we should be concerned about things like efficiency, and speed, and memory when we are building larger applications. Let’s examine a couple of data points. Here I’m using sys.getsizeof to get a fairly decent idea of memory usage.

sys.getsizeof(A)
920
sys.getsizeof(B)
87632

not surprisingly, B > A. Reasonable given that B has 10000 items while A has 5. But think about your application and how long A or B are going to hang around. Are you using them once or many times? Are you losing available memory for a one time operation or do you need to access that iterable again and again? If you only need it once, it is costly to make the iterable and then have it hang around.

Generators

Welcome the generator that will not store the values but rather the method used to compute the values. Let’s create a generator that gives the same result as B above.

    C = (x for x in range(1,10000))

The difference in the definition is the use of () rather than the list comprehension []. What did we get? Let’s look at the differences.

type(A)
< class ‘list’>
type(B)
< class ‘list’>
type(C)
< class ‘generator’>

But try using C in your print loop and you will get the same result! Size differences?

sys.getsizeof(A)
920
sys.getsizeof(B)
87632
sys.getsizeof(C)
80

Now that’s interesting! not only is C 0.09% the size of B, both giving the same result when used for computation, but C is 8.7% the size of A, which only has 5 values compared to C’s 10000 (0.05% of 10000)!

I got the output I wanted from C with a similar iteration, namely:

    for c in C:
        print(c)

and, sure enough, I have 10000 values in output. But when I ran it a second time, I got no output! That’s because my generator was consumed when I iterated over it. They can only be used once. So there are many times when I only need to make the iterable and run over it once in flow, and so that’s fine. And I gain considerable efficiencies.

A more complex example

And these are merely simple iterables. Imagine what happens with more complex structures, e.g.

    letters = ['a', 'b', 'c', 'd', 'e']
    colors = ['red', 'yellow', 'blue', 'green']
    squares = [x*x for x in range(1,10)]

    newlist = [(l, c, s) for l in letters for c in colors for s in squares]
    newgen = ((l, c, s) for l in letters for c in colors for s in squares)

Both newgen and newlist produce objects that can yield a 180 element group of tuples, e.g.:

(‘a’, ‘red’, 1)
(‘a’, ‘red’, 4)
(‘a’, ‘red’, 9)
(‘a’, ‘red’, 16)
(‘a’, ‘red’, 25)
(‘a’, ‘red’, 36)
(‘a’, ‘red’, 49)
(‘a’, ‘red’, 64)
(‘a’, ‘red’, 81)
(‘a’, ‘yellow’, 1)
(‘a’, ‘yellow’, 4)
(‘a’, ‘yellow’, 9)
(‘a’, ‘yellow’, 16)
(‘a’, ‘yellow’, 25)
(‘a’, ‘yellow’, 36)
(‘a’, ‘yellow’, 49)
(‘a’, ‘yellow’, 64)
(‘a’, ‘yellow’, 81)
(‘a’, ‘blue’, 1)
(‘a’, ‘blue’, 4)
(‘a’, ‘blue’, 9)
(‘a’, ‘blue’, 16)
(‘a’, ‘blue’, 25)
(‘a’, ‘blue’, 36)
(‘a’, ‘blue’, 49)
(‘a’, ‘blue’, 64)
(‘a’, ‘blue’, 81)
(‘a’, ‘green’, 1)
(‘a’, ‘green’, 4)
(‘a’, ‘green’, 9)
(‘a’, ‘green’, 16)
(‘a’, ‘green’, 25)
(‘a’, ‘green’, 36)
(‘a’, ‘green’, 49)
(‘a’, ‘green’, 64)
(‘a’, ‘green’, 81)
(‘b’, ‘red’, 1)
(‘b’, ‘red’, 4)
(‘b’, ‘red’, 9)
(‘b’, ‘red’, 16)
(‘b’, ‘red’, 25)
(‘b’, ‘red’, 36)
(‘b’, ‘red’, 49)
(‘b’, ‘red’, 64)
(‘b’, ‘red’, 81)
(‘b’, ‘yellow’, 1)
(‘b’, ‘yellow’, 4)
(‘b’, ‘yellow’, 9)
(‘b’, ‘yellow’, 16)
(‘b’, ‘yellow’, 25)
(‘b’, ‘yellow’, 36)
(‘b’, ‘yellow’, 49)
(‘b’, ‘yellow’, 64)
(‘b’, ‘yellow’, 81)
(‘b’, ‘blue’, 1)
(‘b’, ‘blue’, 4)
(‘b’, ‘blue’, 9)
(‘b’, ‘blue’, 16)
(‘b’, ‘blue’, 25)
(‘b’, ‘blue’, 36)
(‘b’, ‘blue’, 49)
(‘b’, ‘blue’, 64)
(‘b’, ‘blue’, 81)
(‘b’, ‘green’, 1)
(‘b’, ‘green’, 4)
(‘b’, ‘green’, 9)
(‘b’, ‘green’, 16)
(‘b’, ‘green’, 25)
(‘b’, ‘green’, 36)
(‘b’, ‘green’, 49)
(‘b’, ‘green’, 64)
(‘b’, ‘green’, 81)
(‘c’, ‘red’, 1)
(‘c’, ‘red’, 4)
(‘c’, ‘red’, 9)
(‘c’, ‘red’, 16)
(‘c’, ‘red’, 25)
(‘c’, ‘red’, 36)
(‘c’, ‘red’, 49)
(‘c’, ‘red’, 64)
(‘c’, ‘red’, 81)
(‘c’, ‘yellow’, 1)
(‘c’, ‘yellow’, 4)
(‘c’, ‘yellow’, 9)
(‘c’, ‘yellow’, 16)
(‘c’, ‘yellow’, 25)
(‘c’, ‘yellow’, 36)
(‘c’, ‘yellow’, 49)
(‘c’, ‘yellow’, 64)
(‘c’, ‘yellow’, 81)
(‘c’, ‘blue’, 1)
(‘c’, ‘blue’, 4)
(‘c’, ‘blue’, 9)
(‘c’, ‘blue’, 16)
(‘c’, ‘blue’, 25)
(‘c’, ‘blue’, 36)
(‘c’, ‘blue’, 49)
(‘c’, ‘blue’, 64)
(‘c’, ‘blue’, 81)
(‘c’, ‘green’, 1)
(‘c’, ‘green’, 4)
(‘c’, ‘green’, 9)
(‘c’, ‘green’, 16)
(‘c’, ‘green’, 25)
(‘c’, ‘green’, 36)
(‘c’, ‘green’, 49)
(‘c’, ‘green’, 64)
(‘c’, ‘green’, 81)
(‘d’, ‘red’, 1)
(‘d’, ‘red’, 4)
(‘d’, ‘red’, 9)
(‘d’, ‘red’, 16)
(‘d’, ‘red’, 25)
(‘d’, ‘red’, 36)
(‘d’, ‘red’, 49)
(‘d’, ‘red’, 64)
(‘d’, ‘red’, 81)
(‘d’, ‘yellow’, 1)
(‘d’, ‘yellow’, 4)
(‘d’, ‘yellow’, 9)
(‘d’, ‘yellow’, 16)
(‘d’, ‘yellow’, 25)
(‘d’, ‘yellow’, 36)
(‘d’, ‘yellow’, 49)
(‘d’, ‘yellow’, 64)
(‘d’, ‘yellow’, 81)
(‘d’, ‘blue’, 1)
(‘d’, ‘blue’, 4)
(‘d’, ‘blue’, 9)
(‘d’, ‘blue’, 16)
(‘d’, ‘blue’, 25)
(‘d’, ‘blue’, 36)
(‘d’, ‘blue’, 49)
(‘d’, ‘blue’, 64)
(‘d’, ‘blue’, 81)
(‘d’, ‘green’, 1)
(‘d’, ‘green’, 4)
(‘d’, ‘green’, 9)
(‘d’, ‘green’, 16)
(‘d’, ‘green’, 25)
(‘d’, ‘green’, 36)
(‘d’, ‘green’, 49)
(‘d’, ‘green’, 64)
(‘d’, ‘green’, 81)
(‘e’, ‘red’, 1)
(‘e’, ‘red’, 4)
(‘e’, ‘red’, 9)
(‘e’, ‘red’, 16)
(‘e’, ‘red’, 25)
(‘e’, ‘red’, 36)
(‘e’, ‘red’, 49)
(‘e’, ‘red’, 64)
(‘e’, ‘red’, 81)
(‘e’, ‘yellow’, 1)
(‘e’, ‘yellow’, 4)
(‘e’, ‘yellow’, 9)
(‘e’, ‘yellow’, 16)
(‘e’, ‘yellow’, 25)
(‘e’, ‘yellow’, 36)
(‘e’, ‘yellow’, 49)
(‘e’, ‘yellow’, 64)
(‘e’, ‘yellow’, 81)
(‘e’, ‘blue’, 1)
(‘e’, ‘blue’, 4)
(‘e’, ‘blue’, 9)
(‘e’, ‘blue’, 16)
(‘e’, ‘blue’, 25)
(‘e’, ‘blue’, 36)
(‘e’, ‘blue’, 49)
(‘e’, ‘blue’, 64)
(‘e’, ‘blue’, 81)
(‘e’, ‘green’, 1)
(‘e’, ‘green’, 4)
(‘e’, ‘green’, 9)
(‘e’, ‘green’, 16)
(‘e’, ‘green’, 25)
(‘e’, ‘green’, 36)
(‘e’, ‘green’, 49)
(‘e’, ‘green’, 64)
(‘e’, ‘green’, 81)

Yet look at the type and size differences:

< class ‘list’> 1680
< class ‘generator’> 80

newlist is 21 times larger than newgen. So, as you write iterable obejcts, especially iterables that are being used for another computation, consider generators!

Install Tor using apt-get on ubuntu- trusty 14.x





tor.mrd—/Users/shawnmehan


I went to turn on Tor on an ubuntu-64 “trusty” 14.x guest and ran into problems. There appears to be a bug
that critically affects the traditional methods using gpg, so here is some information on how to avoid this.


Tor is free software and an open network that helps you defend against traffic analysis, a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security. There is a very neat Python
controller library for use with Tor called stem.

The problem

To install Tor on an ubuntu, the following instructions can be found:

$ sudo nano /etc/apt/sources.list.d/tor_repo.list

and then add the following lines:

deb http://deb.torproject.org/torproject.org trusty main
deb-src http://deb.torproject.org/torproject.org trusty main

The critical problem then occurs as you try to add the appropriate keyring and key used to sign the packages:


$ gpg –keyserver keys.gnupg.net –recv 886DDD89

$ gpg –export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add

There appears to be a bug with gpg and guests involving not correctly resolving DNS for the gpg commands. The symptom that I was getting was

$ gpg –export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add –
usage: gpg [options] [filename]
gpg: can't open `–': No such file or directory

The fail in the pipe is due to there not being anything actually exported by gpg. You can test this with:


$ gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
gpg: WARNING: nothing exported

Solution

So, take another approach. Use different packages to avoid any of this problem.


$ sudo apt-get update

$ sudo apt-get install deb.torproject.org-keyring

$ sudo apt-get install tor

which will get you something like:

The following NEW packages will be installed:
  deb.torproject.org-keyring
0 upgraded, 1 newly installed, 0 to remove and 63 not upgraded.
Need to get 5,268 B of archives.
After this operation, 7,168 B of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  deb.torproject.org-keyring
Install these packages without verification? [y/N] Y
Get:1 http://deb.torproject.org/torproject.org/ trusty/main deb.torproject.org-keyring all 2014.08.31+b1 [5,268 B]
Fetched 5,268 B in 0s (22.3 kB/s)                     
Selecting previously unselected package deb.torproject.org-keyring.
(Reading database ... 128038 files and directories currently installed.)
Preparing to unpack .../deb.torproject.org-keyring_2014.08.31+b1_all.deb ...
Unpacking deb.torproject.org-keyring (2014.08.31+b1) ...
Setting up deb.torproject.org-keyring (2014.08.31+b1) ...
OK

and

    The following extra packages will be installed:
  libseccomp2 tor-geoipdb torsocks
Suggested packages:
  mixmaster torbrowser-launcher socat tor-arm apparmor-utils obfsproxy
  obfs4proxy
The following NEW packages will be installed:
  libseccomp2 tor tor-geoipdb torsocks
0 upgraded, 4 newly installed, 0 to remove and 63 not upgraded.
Need to get 1,707 kB of archives.
After this operation, 8,053 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
WARNING: The following packages cannot be authenticated!
  tor tor-geoipdb
Install these packages without verification? [y/N] Y
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main libseccomp2 amd64 2.1.0+dfsg-1 [34.8 kB]
Get:2 http://deb.torproject.org/torproject.org/ trusty/main tor amd64 0.2.7.6-1~trusty+1 [1,024 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty/universe torsocks amd64 1.3-3 [73.0 kB]
Get:4 http://deb.torproject.org/torproject.org/ trusty/main tor-geoipdb all 0.2.7.6-1~trusty+1 [575 kB]
Fetched 1,707 kB in 2s (749 kB/s)
Selecting previously unselected package libseccomp2:amd64.
(Reading database ... 128043 files and directories currently installed.)
Preparing to unpack .../libseccomp2_2.1.0+dfsg-1_amd64.deb ...
Unpacking libseccomp2:amd64 (2.1.0+dfsg-1) ...
Selecting previously unselected package tor.
Preparing to unpack .../tor_0.2.7.6-1~trusty+1_amd64.deb ...
Unpacking tor (0.2.7.6-1~trusty+1) ...
Selecting previously unselected package torsocks.
Preparing to unpack .../torsocks_1.3-3_amd64.deb ...
Unpacking torsocks (1.3-3) ...
Selecting previously unselected package tor-geoipdb.
Preparing to unpack .../tor-geoipdb_0.2.7.6-1~trusty+1_all.deb ...
Unpacking tor-geoipdb (0.2.7.6-1~trusty+1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libseccomp2:amd64 (2.1.0+dfsg-1) ...
Setting up tor (0.2.7.6-1~trusty+1) ...
Something or somebody made /var/lib/tor disappear.
Creating one for you again.
Something or somebody made /var/log/tor disappear.
Creating one for you again.
 * Starting tor daemon...                                                                                                                  [ OK ] 
Setting up torsocks (1.3-3) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up tor-geoipdb (0.2.7.6-1~trusty+1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...

Now you have Tor installed, just start it.


$ sudo /etc/init.d/tor start

Starting tor daemon...

It also appears that there is some confusion as to the defaul port the service is running on. I found it to be running on 9050, not 9150 as some
articles are reporting. I haven’t yet found an easy way to determine this but trial-and-error proved it to be true.


HTML5 Mathematical Entities

Mathematical HTML entities

The following is a list of all the mathematical entities available under compliant HTML5. It is meant as a reference. Where available, the entity is given, but all entities have their decimal and hexidecimal reference listed.

Char Dec Hex Entity Name
8704 2200 &forall; FOR ALL
8705 2201   COMPLEMENT
8706 2202 &part; PARTIAL DIFFERENTIAL
8707 2203 &exist; THERE EXISTS
8708 2204   THERE DOES NOT EXIST
8709 2205 &empty; EMPTY SET
8710 2206   INCREMENT
8711 2207 &nabla; NABLA
8712 2208 &isin; ELEMENT OF
8713 2209 &notin; NOT AN ELEMENT OF
8714 220A   SMALL ELEMENT OF
8715 220B &ni; CONTAINS AS MEMBER
8716 220C   DOES NOT CONTAIN AS MEMBER
8717 220D   SMALL CONTAINS AS MEMBER
8718 220E   END OF PROOF
8719 220F &prod; N-ARY PRODUCT
8720 2210   N-ARY COPRODUCT
8721 2211 &sum; N-ARY SUMMATION
8722 2212 &minus; MINUS SIGN
8723 2213   MINUS-OR-PLUS SIGN
8724 2214   DOT PLUS
8725 2215   DIVISION SLASH
8726 2216   SET MINUS
8727 2217 &lowast; ASTERISK OPERATOR
8728 2218   RING OPERATOR
8729 2219   BULLET OPERATOR
8730 221A &radic; SQUARE ROOT
8731 221B   CUBE ROOT
8732 221C   FOURTH ROOT
8733 221D &prop; PROPORTIONAL TO
8734 221E &infin; INFINITY
8735 221F   RIGHT ANGLE
8736 2220 &ang; ANGLE
8737 2221   MEASURED ANGLE
8738 2222   SPHERICAL ANGLE
8739 2223   DIVIDES
8740 2224   DOES NOT DIVIDE
8741 2225   PARALLEL TO
8742 2226   NOT PARALLEL TO
8743 2227 &and; LOGICAL AND
8744 2228 &or; LOGICAL OR
8745 2229 &cap; INTERSECTION
8746 222A &cup; UNION
8747 222B &int; INTEGRAL
8748 222C   DOUBLE INTEGRAL
8749 222D   TRIPLE INTEGRAL
8750 222E   CONTOUR INTEGRAL
8751 222F   SURFACE INTEGRAL
8752 2230   VOLUME INTEGRAL
8753 2231   CLOCKWISE INTEGRAL
8754 2232   CLOCKWISE CONTOUR INTEGRAL
8755 2233   ANTICLOCKWISE CONTOUR INTEGRAL
8756 2234 &there4; THEREFORE
8757 2235   BECAUSE
8758 2236   RATIO
8759 2237   PROPORTION
8760 2238   DOT MINUS
8761 2239   EXCESS
8762 223A   GEOMETRIC PROPORTION
8763 223B   HOMOTHETIC
8764 223C &sim; TILDE OPERATOR
8765 223D   REVERSED TILDE
8766 223E   INVERTED LAZY S
8767 223F   SINE WAVE
8768 2240   WREATH PRODUCT
8769 2241   NOT TILDE
8770 2242   MINUS TILDE
8771 2243   ASYMPTOTICALLY EQUAL TO
8772 2244   NOT ASYMPTOTICALLY EQUAL TO
8773 2245 &cong; APPROXIMATELY EQUAL TO
8774 2246   APPROXIMATELY BUT NOT ACTUALLY EQUAL TO
8775 2247   NEITHER APPROXIMATELY NOR ACTUALLY EQUAL TO
8776 2248 &asymp; ALMOST EQUAL TO
8777 2249   NOT ALMOST EQUAL TO
8778 224A   ALMOST EQUAL OR EQUAL TO
8779 224B   TRIPLE TILDE
8780 224C   ALL EQUAL TO
8781 224D   EQUIVALENT TO
8782 224E   GEOMETRICALLY EQUIVALENT TO
8783 224F   DIFFERENCE BETWEEN
8784 2250   APPROACHES THE LIMIT
8785 2251   GEOMETRICALLY EQUAL TO
8786 2252   APPROXIMATELY EQUAL TO OR THE IMAGE OF
8787 2253   IMAGE OF OR APPROXIMATELY EQUAL TO
8788 2254   COLON EQUALS
8789 2255   EQUALS COLON
8790 2256   RING IN EQUAL TO
8791 2257   RING EQUAL TO
8792 2258   CORRESPONDS TO
8793 2259   ESTIMATES
8794 225A   EQUIANGULAR TO
8795 225B   STAR EQUALS
8796 225C   DELTA EQUAL TO
8797 225D   EQUAL TO BY DEFINITION
8798 225E   MEASURED BY
8799 225F   QUESTIONED EQUAL TO
8800 2260 &ne; NOT EQUAL TO
8801 2261 &equiv; IDENTICAL TO
8802 2262   NOT IDENTICAL TO
8803 2263   STRICTLY EQUIVALENT TO
8804 2264 &le; LESS-THAN OR EQUAL TO
8805 2265 &ge; GREATER-THAN OR EQUAL TO
8806 2266   LESS-THAN OVER EQUAL TO
8807 2267   GREATER-THAN OVER EQUAL TO
8808 2268   LESS-THAN BUT NOT EQUAL TO
8809 2269   GREATER-THAN BUT NOT EQUAL TO
8810 226A   MUCH LESS-THAN
8811 226B   MUCH GREATER-THAN
8812 226C   BETWEEN
8813 226D   NOT EQUIVALENT TO
8814 226E   NOT LESS-THAN
8815 226F   NOT GREATER-THAN
8816 2270   NEITHER LESS-THAN NOR EQUAL TO
8817 2271   NEITHER GREATER-THAN NOR EQUAL TO
8818 2272   LESS-THAN OR EQUIVALENT TO
8819 2273   GREATER-THAN OR EQUIVALENT TO
8820 2274   NEITHER LESS-THAN NOR EQUIVALENT TO
8821 2275   NEITHER GREATER-THAN NOR EQUIVALENT TO
8822 2276   LESS-THAN OR GREATER-THAN
8823 2277   GREATER-THAN OR LESS-THAN
8824 2278   NEITHER LESS-THAN NOR GREATER-THAN
8825 2279   NEITHER GREATER-THAN NOR LESS-THAN
8826 227A   PRECEDES
8827 227B   SUCCEEDS
8828 227C   PRECEDES OR EQUAL TO
8829 227D   SUCCEEDS OR EQUAL TO
8830 227E   PRECEDES OR EQUIVALENT TO
8831 227F   SUCCEEDS OR EQUIVALENT TO
8832 2280   DOES NOT PRECEDE
8833 2281   DOES NOT SUCCEED
8834 2282 &sub; SUBSET OF
8835 2283 &sup; SUPERSET OF
8836 2284 &nsub; NOT A SUBSET OF
8837 2285   NOT A SUPERSET OF
8838 2286 &sube; SUBSET OF OR EQUAL TO
8839 2287 &supe; SUPERSET OF OR EQUAL TO
8840 2288   NEITHER A SUBSET OF NOR EQUAL TO
8841 2289   NEITHER A SUPERSET OF NOR EQUAL TO
8842 228A   SUBSET OF WITH NOT EQUAL TO
8843 228B   SUPERSET OF WITH NOT EQUAL TO
8844 228C   MULTISET
8845 228D   MULTISET MULTIPLICATION
8846 228E   MULTISET UNION
8847 228F   SQUARE IMAGE OF
8848 2290   SQUARE ORIGINAL OF
8849 2291   SQUARE IMAGE OF OR EQUAL TO
8850 2292   SQUARE ORIGINAL OF OR EQUAL TO
8851 2293   SQUARE CAP
8852 2294   SQUARE CUP
8853 2295 &oplus; CIRCLED PLUS
8854 2296   CIRCLED MINUS
8855 2297 &otimes; CIRCLED TIMES
8856 2298   CIRCLED DIVISION SLASH
8857 2299   CIRCLED DOT OPERATOR
8858 229A   CIRCLED RING OPERATOR
8859 229B   CIRCLED ASTERISK OPERATOR
8860 229C   CIRCLED EQUALS
8861 229D   CIRCLED DASH
8862 229E   SQUARED PLUS
8863 229F   SQUARED MINUS
8864 22A0   SQUARED TIMES
8865 22A1   SQUARED DOT OPERATOR
8866 22A2   RIGHT TACK
8867 22A3   LEFT TACK
8868 22A4   DOWN TACK
8869 22A5 &perp; UP TACK
8870 22A6   ASSERTION
8871 22A7   MODELS
8872 22A8   TRUE
8873 22A9   FORCES
8874 22AA   TRIPLE VERTICAL BAR RIGHT TURNSTILE
8875 22AB   DOUBLE VERTICAL BAR DOUBLE RIGHT TURNSTILE
8876 22AC   DOES NOT PROVE
8877 22AD   NOT TRUE
8878 22AE   DOES NOT FORCE
8879 22AF   NEGATED DOUBLE VERTICAL BAR DOUBLE RIGHT TURNSTILE
8880 22B0   PRECEDES UNDER RELATION
8881 22B1   SUCCEEDS UNDER RELATION
8882 22B2   NORMAL SUBGROUP OF
8883 22B3   CONTAINS AS NORMAL SUBGROUP
8884 22B4   NORMAL SUBGROUP OF OR EQUAL TO
8885 22B5   CONTAINS AS NORMAL SUBGROUP OR EQUAL TO
8886 22B6   ORIGINAL OF
8887 22B7   IMAGE OF
8888 22B8   MULTIMAP
8889 22B9   HERMITIAN CONJUGATE MATRIX
8890 22BA   INTERCALATE
8891 22BB   XOR
8892 22BC   NAND
8893 22BD   NOR
8894 22BE   RIGHT ANGLE WITH ARC
8895 22BF   RIGHT TRIANGLE
8896 22C0   N-ARY LOGICAL AND
8897 22C1   N-ARY LOGICAL OR
8898 22C2   N-ARY INTERSECTION
8899 22C3   N-ARY UNION
8900 22C4   DIAMOND OPERATOR
8901 22C5 &sdot; DOT OPERATOR
8902 22C6   STAR OPERATOR
8903 22C7   DIVISION TIMES
8904 22C8   BOWTIE
8905 22C9   LEFT NORMAL FACTOR SEMIDIRECT PRODUCT
8906 22CA   RIGHT NORMAL FACTOR SEMIDIRECT PRODUCT
8907 22CB   LEFT SEMIDIRECT PRODUCT
8908 22CC   RIGHT SEMIDIRECT PRODUCT
8909 22CD   REVERSED TILDE EQUALS
8910 22CE   CURLY LOGICAL OR
8911 22CF   CURLY LOGICAL AND
8912 22D0   DOUBLE SUBSET
8913 22D1   DOUBLE SUPERSET
8914 22D2   DOUBLE INTERSECTION
8915 22D3   DOUBLE UNION
8916 22D4   PITCHFORK
8917 22D5   EQUAL AND PARALLEL TO
8918 22D6   LESS-THAN WITH DOT
8919 22D7   GREATER-THAN WITH DOT
8920 22D8   VERY MUCH LESS-THAN
8921 22D9   VERY MUCH GREATER-THAN
8922 22DA   LESS-THAN EQUAL TO OR GREATER-THAN
8923 22DB   GREATER-THAN EQUAL TO OR LESS-THAN
8924 22DC   EQUAL TO OR LESS-THAN
8925 22DD   EQUAL TO OR GREATER-THAN
8926 22DE   EQUAL TO OR PRECEDES
8927 22DF   EQUAL TO OR SUCCEEDS
8928 22E0   DOES NOT PRECEDE OR EQUAL
8929 22E1   DOES NOT SUCCEED OR EQUAL
8930 22E2   NOT SQUARE IMAGE OF OR EQUAL TO
8931 22E3   NOT SQUARE ORIGINAL OF OR EQUAL TO
8932 22E4   SQUARE IMAGE OF OR NOT EQUAL TO
8933 22E5   SQUARE ORIGINAL OF OR NOT EQUAL TO
8934 22E6   LESS-THAN BUT NOT EQUIVALENT TO
8935 22E7   GREATER-THAN BUT NOT EQUIVALENT TO
8936 22E8   PRECEDES BUT NOT EQUIVALENT TO
8937 22E9   SUCCEEDS BUT NOT EQUIVALENT TO
8938 22EA   NOT NORMAL SUBGROUP OF
8939 22EB   DOES NOT CONTAIN AS NORMAL SUBGROUP
8940 22EC   NOT NORMAL SUBGROUP OF OR EQUAL TO
8941 22ED   DOES NOT CONTAIN AS NORMAL SUBGROUP OR EQUAL
8942 22EE   VERTICAL ELLIPSIS
8943 22EF   MIDLINE HORIZONTAL ELLIPSIS
8944 22F0   UP RIGHT DIAGONAL ELLIPSIS
8945 22F1   DOWN RIGHT DIAGONAL ELLIPSIS
8946 22F2   ELEMENT OF WITH LONG HORIZONTAL STROKE
8947 22F3   ELEMENT OF WITH VERTICAL BAR AT END OF HORIZONTAL STROKE
8948 22F4   SMALL ELEMENT OF WITH VERTICAL BAR AT END OF HORIZONTAL STROKE
8949 22F5   ELEMENT OF WITH DOT ABOVE
8950 22F6   ELEMENT OF WITH OVERBAR
8951 22F7   SMALL ELEMENT OF WITH OVERBAR
8952 22F8   ELEMENT OF WITH UNDERBAR
8953 22F9   ELEMENT OF WITH TWO HORIZONTAL STROKES
8954 22FA   CONTAINS WITH LONG HORIZONTAL STROKE
8955 22FB   CONTAINS WITH VERTICAL BAR AT END OF HORIZONTAL STROKE
8956 22FC   SMALL CONTAINS WITH VERTICAL BAR AT END OF HORIZONTAL STROKE
8957 22FD   CONTAINS WITH OVERBAR
8958 22FE   SMALL CONTAINS WITH OVERBAR
8959 22FF   Z NOTATION BAG MEMBERSHIP

Git cheatsheet for cli

#Git cheatsheet

Here is a cheatsheet on git.


Create

  • Clone an existing repo
    $ git clone ssh:://user@my.com/repo.git
  • Create a new local repo
    $ git init

Local Changes

  • Changed files in your working directory
    $ git status
  • Changes to tracked files
    $ git diff
  • Add all current changes to the next commit
    $ git add .
  • Add some changes in < file > to the next commit
    $ git add -p < file >
  • Commit all local changes in tracked files
    $ git commit -a
  • Commit previously staged changes
    $ git commit
  • Change the last commit
    $ git commit --amend

Commit History

  • Show all commits, starting with newest
    $ git log
  • Show changes over time for a specific file
    $ git log -p < file >
  • Who changes what and when in < file >
    $ git blame < file >

Branches & Tags

  • List all existing branches
    $ git branch -av
  • Switch HEAD branch
    $ git checkout < branch >
  • Create a new branch based on your current HEAD
    $ git branch < new-branch >
  • Create a new tracking branch based on a remote branch
    $ git checkout --track < remote/branch >
  • Delete a local branch
    $ git branch -d < branch >
  • Mark the current commit with a tag
    $ git tag < tag-name >

Update & Publish

  • List all currently configured remotes
    $ git remote -v
  • Show information about a remote
    $ git remote show < remote >
  • Add new remote repository, named < remote >
    $ git remote add < shortname > < url >
  • Download all changes from < remote >, but don't integrate into HEAD
    $ git fetch < remote >
  • Download changes and directly merge/integrate into HEAD
    $ git pull < remote > < branch >
  • Publish local changes on a remote
    $ git push < remote > < branch >
  • Delete a branch on the remote
    $ git branch -dr < remote/branch >
  • Publish your tags
    $ git push --tags

Merge & Rebase

  • Merge < branch > into your current HEAD
    $ git merge < branch >
  • Rebase your current HEAD onto < branch >
    $ git rebase < branch >
  • Abort a rebase
    $ git rebase --abort
  • Continue a rebase after resolving conflicts
    $ git rebase --continue
  • Use your configured merge tool to solve conflicts
    $ git mergetool
  • Use your editor to manually solve conflicts and (after resolving) mark file as resolved
    $ git add < resolved-file >
    $ git rm < resolved-file >

Undo

  • Discard all local changes in your working directory
    $ git reset --hard HEAD
  • Discard local changes in a specific file
    $ git checkout HEAD < file >
  • Revert a commit (by producing a new commit with contrary changes)
    $ git revert < commit >
  • Reset your HEAD pointer to a previous commit and discard all changes since
    $ git reset --hard < commit >
  • and preserve all changes as unstaged changes
    $ git reset < commit >
  • and preserve uncommitted local changes
    $ git reset --keep < commit >

Learn git

Yahoo Financial data API

Access financial data from web api at yahoo

Yahoo used to run a very rich API to financial data at , but, alas, it serves no more on most of the URLs. There is still a service, but it
is a pale shadow of the former. The former URLs had up to 84 parameters! (see the reference of the API at the bottom). Now
you can query


http://real-chart.finance.yahoo.com/table.csv?s=AAPL

and you will get a return of Apple’s data, going back for decades:


Date,Open,High,Low,Close,Volume,Adj Close

It’s all comma delimited, and there doesn’t appear to be any secret xml switch. This is all dumbed down from what used to be there.
You can get the same result from another deprecated service they had at
http://ichart.finance.yahoo.com/table.csv?s=GOOG, again replacing
your S param with the symbol you wish to look up. No other params seem to make any difference to the request or will
result in a 404. One doesn’t seem to be able to pull multiple symbols in a single query.

There are other services that still seem to be running:

Download your CSV

http://download.finance.yahoo.com/d/quotes.csv?s=AAPL+GOOG&f=snl1c1p2&e=.csv

which will give you a downloaded CSV with quotes in the form of:


"AAPL","Apple Inc.",112.72,-0.20,"-0.18%"
"GOOG","Google Inc.",628.59,-9.02,"-1.41%"

This is accepting multiple symbols. It also seems to be accepting many of the old parameters referenced below.
Perhaps this should be reworked into a little api as it’s pretty hairy just now. I might do that in R to see how useful
it could be for time series. the extension makes no difference to what the format is nor the actual file sent. So,

  • s = <+SYM>…
  • f =
  • e no difference

Want a chart?

You can request a symbol at http://chart.finance.yahoo.com/z?s=GOOG
and you will get Google stock chart

Want a snapshot, no historical, in XML?

You can still reach through to the backend through this quite amazing piece of internet fossil evidence.
http://query.yahooapis.com/v1/public/yql?q=select%20%20from%20yahoo.finance.quotes%20where%20symbol%20in%20%28%22AAPL,GOOG%22%29&env=store://datatables.org/alltableswithkeys
will yield a *mostly
empty XML block. But, hey, it’s there.

Need a symbol, go here.

Parameter API

  • a Ask
  • a2 Average Daily Volume
  • a5 Ask Size
  • b Bid
  • b2 Ask (Real-time)
  • b3 Bid (Real-time)
  • b4 Book Value
  • b6 Bid Size
  • c Change & Percent Change
  • c1 Change
  • c3 Commission
  • c6 Change (Real-time)
  • c8 After Hours Change (Real-time)
  • d Dividend/Share
  • d1 Last Trade Date
  • d2 Trade Date
  • e Earnings/Share
  • e1 Error Indication (returned for symbol changed / invalid)
  • e7 EPS Estimate Current Year
  • e8 EPS Estimate Next Year
  • e9 EPS Estimate Next Quarter
  • f6 Float Shares
  • g Day’s Low
  • h Day’s High
  • j 52-week Low
  • k 52-week High
  • g1 Holdings Gain Percent
  • g3 Annualized Gain
  • g4 Holdings Gain
  • g5 Holdings Gain Percent (Real-time)
  • g6 Holdings Gain (Real-time)
  • i More Info
  • i5 Order Book (Real-time)
  • j1 Market Capitalization
  • j3 Market Cap (Real-time)
  • j4 EBITDA
  • j5 Change From 52-week Low
  • j6 Percent Change From 52-week Low
  • k1 Last Trade (Real-time) With Time
  • k2 Change Percent (Real-time)
  • k3 Last Trade Size
  • k4 Change From 52-week High
  • k5 Percebt Change From 52-week High
  • l Last Trade (With Time)
  • l1 Last Trade (Price Only)
  • l2 High Limit
  • l3 Low Limit
  • m Day’s Range
  • m2 Day’s Range (Real-time)
  • m3 50-day Moving Average
  • m4 200-day Moving Average
  • m5 Change From 200-day Moving Average
  • m6 Percent Change From 200-day Moving Average
  • m7 Change From 50-day Moving Average
  • m8 Percent Change From 50-day Moving Average
  • n Name
  • n4 Notes
  • o Open
  • p Previous Close
  • p1 Price Paid
  • p2 Change in Percent
  • p5 Price/Sales
  • p6 Price/Book
  • q Ex-Dividend Date
  • r P/E Ratio
  • r1 Dividend Pay Date
  • r2 P/E Ratio (Real-time)
  • r5 PEG Ratio
  • r6 Price/EPS Estimate Current Year
  • r7 Price/EPS Estimate Next Year
  • s Symbol
  • s1 Shares Owned
  • s7 Short Ratio
  • t1 Last Trade Time
  • t6 Trade Links
  • t7 Ticker Trend
  • t8 1 yr Target Price
  • v Volume
  • v1 Holdings Value
  • v7 Holdings Value (Real-time)
  • w 52-week Range
  • w1 Day’s Value Change
  • w4 Day’s Value Change (Real-time)
  • x Stock Exchange
  • y Dividend Yield

R Data Structures

R Data Structures overview by Hadley Wickham

If you are working with any programming language, there is nothing more important to understand fundamentally than
the language’s underlying data structures. Wickham on R Data Structures is an
excellent overview for R programmers.


There are five fundamental data types in R.

  • Homogeneous
    1. 1D – Atomic vector
    2. 2D – Matrix
    3. nD – Array
  • Heterogeneous
    1. List
    2. Data frame

Hadley goes through the five to show how they compare, contrast and, most importantly, they are interrelated. Important stuff.
He also goes through a small set of exercises to test comprehension. I think that some of these could be used
as bones for interview questions.

Taken from his book, Advanced R which
is well worth the price and should be read by serious R folks.

Sublime for blogging in WP

Reducing Friction in WordPress blogging

So, I just found a nice post on zero-friction blogging using the all good Sublime text editor.
Which is a real treat, because I used to use TextMate directly into WP, but that went away
a couple of years ago with the demise of TextMate and the wierdness of WP. But, take heart!
Sublime continues to amaze me as an even better editor than TextMate, with so
much Emacs power in it. And now the plugin community has really ponied up and made
some nice plugins that can greatly reduce friction in your workflow for making posts into WP.


Use Markdown again!

Markdown is back in my workflow. For the last few months I’ve been using the WP admin
blog editor. Even with the dreaded WYSIWYG turned off, it still pretty much sucks.
Markdown is so much easier.
I just couldn’t find a replacement for the markdown plugin that was working fine in
my old version of TextMate.

So, assuming that you have already installed Sublime then you
need to install Package Control if you haven’t already.
Also, you might want to learn about some of the awesome sauce you get in Sublime. Once you have those done, you need to open
up the Package Installer in Sublime {super}+{shift}+{p} and install the following:

Create in Sublime MD

Go into Sublime with those plugins. Create a new MD file. Sublime is awesome in that it is
contextually aware of file type and will load a whole slew of macros and functions relevant to
that context. MD tells it that it’s markdown, and you’ll get all of that sugar to drive your post.

Once you have it to where you think your’re good, use the OmniMarkupPreviewer to see how
it’s going to look as HTML, which is what you’re going to output it to in order to make
it nice for WP to consume. You can do that with {alt}+{super}+{O} which will cleverly force the
HTML to be rendered by your default browser. When you’re happy with that render, use
Markdown to Clipboard to make a clean HTML export via {right-click} and paste that
into your new post in WP.

This isn’t as straight as just uploading it, but it still gives you Markdown and all of the goodness
of the Sublime editor, plus the nice UX of the markdown editing theme. We might yet
be able to get it to push into WP, I’ll be looking into that. I saw that there were some
older (circ 2014) plugins for WP that might be alright or at least updateable.

Code syntax tool for R into HTML

So this is useful for R people. Need to place some code up in HTML, want to have syntax highlighting, don’t want to fight code and pre format tags all day in wordpress? Paste your block into pretty-r:

data(tips, package="reshape2")
 
tipsAnova <- aov(tip~day-1, data=tips)
tipsLM <- lm(tip~day-1, data=tips)
summary(tipsAnova)
summary(tipsLM)

What this is doing is allowing you to maintain syntactically highlighted and well-formed R source code in your HTML pages, easily. You have already written the source in R, cut and paste it into the form, and it will return a clean set of styled HTML for pasting into any web source you need.