git rebase and filter branch

git rebase when email address is incorrect
git log: shows the email addresses are not the git anonymous address
Fix the ones before push by using git rebase -i

Add the following after each pick xxxxx line using vim editor
exec git commit --amend --author="dougc333 <124989 dougc333="" users.noreply.github.com=""> -C HEAD""

Then git will cycle through each entry in the rebase -i display and you write quit each one wo modifying anything. Then you can push with updated anonymous email.

for entries already committed do a git filter branch.

For path with spaces add a backslash git filter-branch -f --tree-filter 'rm -f fabinger_2022/Untitled\ Folder/pbmc_1k_v3_raw_feature_bc_matrix.h5' HEAD --all
Don't add the -r recursive flag to rm -f fabigner_2022/... This can cause an error message "Cannot rewrite branches: You have unstaged changes"

itertools product 3x faster than list comprehension

>>> timeit.timeit('list(itertools.product([1,2,3,4],["a","b","c","d"]))',number=100)
0.00015309499997329112
>>> timeit.timeit('[(x,y) for i,x in enumerate([1,2,3,4]) for j,y in enumerate(["a","b","c","d"])]',number=100)
0.00034536499998694126
>>>

python module notes

Each OS/Linux distribution has a combination of different install systems including apt, rpm, pip, conda, homebrew. Each install system consists of a central repo to distribute packages and a format for a config file to run both system OS commands, shell scripts and code; where code is some form of standard programming language for additional flexibility. 

There are multiple ways to install python programs/modules, apt install, pip install, conda install, brew install. 

After installation the expectation is import xxx can be used and a CLI if there is one. 

sys.path + PYTHON_PATH environment variables for imp module to look for package_name using import package_name. imp.find_module('torch') to find the module path. 

pip install: installs to directory prefix/site-packages

apt install: installs to directory prefix/dist-packages

distutils was the original python2.x specification using setup.py to install to a site-packages directory. This has been replaced by distutils2 which has been replaced by pip. The original convention of setup.py and site-packages still holds but has been supplemented by setuptools. Setuptools allows a python function specified by module path, module file and function name to be the executable command.

virtual environment site-packages:like MACOS and everybody else there is a semi global path for the anaconda env and one for the user level installed venvs in another scope. 

/anaconda3/lib/pythonX/.../site-packages

/anaconda3/venv/pythonX/.../site-packages

MACOS site-packages: global python install /Libraray/Python/2.7/site-packages; needed for MACOS python system functionality. 

MACOS site-packages: another python install at /usr/local/Cellar/xxx/site-packages for homebrew. User installed packages

To package a python module, we specify an executable name for the program and a function which is run when the executable is typed in the command line. For example foo would run function main under file name foo_file under directory foo_module would have 'console_scripts':['foo=foo_module.foo_file.main']

Use python modules setuptools and click. The setuptools entry_points parameter is a user configurable CLI.

entry_points={
'console_scripts':['executable_name=directory.module_file.py name.function_name']
}


Not all executable names work, aaa has to be an available name and not used for another program. You may not get an error message in the case of conflict. find_packages can be used instead of manually specifying packages which is a list of all the directories you want to include:

from setuptools import setup, find_packages

setup(

name='test_py',

version='0.1.0',

packages=['pt'],

install_requires=[

'Click',

],

entry_points={

'console_scripts': [

'aaa = pt.file_name:main',

],

},

)

pt
  setup.py
  pt
  __init__.py
  file_name.py


>pip install --editable . in the directory where setup.py is located

@click.option, @click.pass_context,

Train a model:
Predict a data sample:

macbook transfer imessage to new computer

The contents of iMessage are stored under ~/Library/Messages. This is NOT the same as /Library/Messages. Very confusing. Note the relative path. If you are in the right subdirectory the Messages folder will look like:





You can copy the entire contents of the folder to your new macbook under ~/Library/Messages which will restore everything including the attachments and archived files. Is easier to copy only the 3 files .db, .shm and .wal which contain the messages and headers for contacts. Restart to get the old messages to show up. Phone numbers may not have contacts associated with them.

You can read the .db file with sqlite3.

changing mac terminal command prompt

The mac terminal prompt is ComputerName: CurrentPath
You will only see the terminal prompt change after a restart.
To change ComputerName:

1. Open a terminal.
2. Type the following command to change the primary hostname of your Mac:
   This is your fully qualified hostname, for example myMac.domain.com
   sudo scutil --set HostName
3. Type the following command to change the Bonjour hostname of your Mac:
   This is the name usable on the local network, for example myMac.local.
   sudo scutil --set LocalHostName
4. Optional: If you also want to change the computer name, type the following command:
   This is the user-friendly computer name you see in Finder, for example myMac.
   sudo scutil --set ComputerName
5. Flush the DNS cache by typing:
   dscacheutil -flushcache
6. Restart your Mac.

how to get files off coursera jupyter notebook

The Coursera VMs dont allow you to install ssh or scp for security reasons. You can email files off the website in python.


import smtplib

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart

import os

def mail_stuff():
    """
    mail stuff to gmail account cause you can't do ssh or scp from coursera
    """
    user_id="xxx"
    password = "xxx"
 
    outer = MIMEMultipart()
    outer['Subject'] = 'files from coursera'
    outer['To'] = "dougchang25@gmail.com"
    outer['From'] = "dougchang25@gmail.com"
    outer.preamble = 'You will not see this in a MIME-aware mail reader.\n'
    attachments = ['images/classification_kiank.png']
    for file in attachments:
        try:
            with open(file, 'rb') as fp:
                msg = MIMEBase('application', "octet-stream")
                msg.set_payload(fp.read())
            encoders.encode_base64(msg)
            msg.add_header('Content-Disposition', 'attachment', filename=os.path.basename(file))
            outer.attach(msg)
        except:
            print("Unable to open one of the attachments. Error: ", sys.exc_info()[0])
            raise

    composed = outer.as_string()

 
 
    try:
        server = smtplib.SMTP_SSL('smtp.gmail.com', 465)
        server.ehlo()
        server.login(user_id, password)
        server.sendmail(user_id, user_id, composed)
        server.close()
        print("sent!!!!")
    except:
        print ('Something went wrong...')
     
mail_stuff()

fastai installation notes

install anaconda3 for python3.6. Make sure you don't have /usr/bin/python pointing to python2.7.

This is the default setting and you have to remove the symlink or you can leave it but make sure the PATH environment variable has /home/username/anaconda3/bin/python set first so the python version from here is found first. Easiest to delete the symlink.

when using conda env update, it creates the fastai conda env based on environment.yml. It does not install keras or tensorflow; you have to pick pip install tensorflow for CPU or pip install tensorflow-gpu if you have a gpu installed. JH's examples are designed for GPU.