Snip
Snip is a utility that I created to store personal data such as articles, code snippets, and reference material. Each document is stored in SQLite and analyzed to produce a document term matrix to allow very fast full text search. I have been consistently disappointed with proprietary note applications as they often store data in proprietary or inconvenient formats that are not easy to export and have no ability to interact with a UNIX shell.
For my needs, an ideal data tool should be designed interface well with a shell, as that is the primary interface that most of my work revolves around. I need to be able to pipe data into and out of storage for quick reference or to easily migrate to other tools. Using standard input and output satisfies most of these criteria, and allows for easy export or import from the clipboard. I also added the ability to attach binary files to these text documents using a BLOB data type.
Full text search is implemented by splitting words based on the unicode standard, then stemming all terms via the Martin Porter method. Indexing is performed upon document creation and update, but all documents can be re-indexed by using the index subcommand. This allows for a fast reference to all documents containing the requested search terms along with unions, intersections, and exclusions.
Data storage defaults to the $HOME directory in a file named .snip.sqlite3. This location can be customized by assigning $SNIP_DB to the full file path of your choosing, which will be honored when snip is invoked. This method also affords a simple way to use different databases to isolate data.
export SNIP_DB=/my/custom/location/database.sqlite3Subcommands
- add - add a new document
- ls - list documents contained in database
- get - output the document source text
- rename - rename a document referenced by uuid
- update - update the contents of a document
- rm - remove a document
- attach - add or retrieve a binary attachment
- search - search all documents for supplied terms
add
Documents can be added by piping text to standard input or by reading a specified file with the -f option. Names are automatically generated by starting text or can be specified by using the -n option.
Add new snip to database
Usage: snip add [OPTIONS]
Options:
-f, --file <file> document text from file
-n, --name <name> name of new document
-v verbose (pass output to stdout)
-h, --help Print help
# add a text file directly by name with autogen title
snip add -f my_file.txt
# specify a document title at creation
snip add -n "Specified Document Title" -f my_file.txt
# add a document via standard input
cat my_interesting_data.txt | snip add
ls
List documents contained in the database
List snips
Usage: snip ls [OPTIONS]
Options:
-l display full uuid
-n <number> number of documents to list
-s display size in bytes
-t display timestamp
-h, --help Print help
# list all documents in database
snip ls
# list documents showing timestamp metadata
snip ls -t
# list the most recent 10 documents
snip ls -n 10
get
Documents are assigned a UUID and can be referenced by a partial hash for brevity. Search is performed based on terms supplied, and each result is provided with context. The raw text of a document can be printed to standard output using the --raw option. Otherwise, the output will include document metadata.
Get from uuid
Usage: snip get [OPTIONS] [uuid]
Arguments:
[uuid]
Options:
-a, --analyze print analyzed document text
-r, --raw print raw document text only (no headers)
-h, --help Print help
# add a text file directly by name with autogen title
snip get 7fa8c4
# raw output with no metadata
snip get --raw 7fa8c4
rename
Rename document
Usage: snip rename [uuid] [name]
Arguments:
[uuid] partial/full id of document
[name] new name
Options:
-h, --help Print help
# rename an existing document
snip rename 7fa8c4 "New Document Title"
update
Updating a document is done by using the update subcommand and the -f option to specify an input file. If the UUID in the supplied file metadata matches, the contents of the document will be updated in the database. The verbose option will print the newly updated data.
Update document from modified file
Usage: snip update [OPTIONS] <file>...
Arguments:
<file>... edited document file
Options:
-r remove document file on successful update
-h, --help Print help
rm
Remove a document. All associated binary attachments will also be removed.
Remove items
Usage: snip rm <ids>...
Arguments:
<ids>...
Options:
-h, --help Print help
# Remove documents from database (also removes associated attachments)
snip rm 7fa8c4
attach
Attach binary data to document
Usage: snip attach <COMMAND>
Commands:
add add file to document
ls list attachments
rm remove attachments
write write attachment to local file
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
search
Search for terms within all documents
Usage: snip search [OPTIONS] <terms>...
Arguments:
<terms>...
Options:
-x, --exclude <exclude> exclude these comma delineated terms
-c, --count print match count only (no excerpts)
-C, --context <context> number of surrounding context words displayed
--excerpts <excerpt-limit> limit the number of match excerpts displayed
-l, --limit <limit> max number of documents to match
--raw do not strip newlines or returns from search excerpt
-u, --uuid <uuid> search for matches in specified documents only
-h, --help Print help
