Where Bookends rocks

Before, I wrote a few complains (https://dellu.wordpress.com/2016/03/28/where-bookends-sucks/) I have on Bookends as replacement for Sente. It has been almost 9 months since that post.

Now, I am updating my experience with the Bookends.

The good news: the latest version of Bookends (12.7.7) has solved one of the major complains I had on the application. That is: it can now extract references from a plethora of sources that the source of extraction is no more a problem.

In addition, I learned in the course of the previous months that the right method of finding references of PDF books is to use the ISBN associated with the books. Once you write the ISBN of the book in the ISBN field, Bookends happily can download all the reference data from Amazon using its Autofill feature. The Autofill feature is a super-feature. One day,  I put about 400 PDF articles collected over the years into Bookends watch folder; all of them have their DOIs printed in the first pages of the articles. It was just magic to see all the PDF files find their references filled automatically in a couple of minutes.

I also leaned that Bookends is much potent application than Sente when it comes to managing references. The tools embedded into it are unbelievably rich. It has organizational tools like: smart folders which support REGEX,  static folders,  labelling (color coding). It has a fully configurable Format Manager which enables the end user to import or export; rename the references in endless ways. The Format Manager in Bookends is the greatest feature I have ever seen in any reference manager. It is a dream come true. It is a liberating feature. I cannot say enough about it.

Bookends has other tools under Global Change. Finder replace that works across the whole library; batch changing of reference types, batch inserting data..and, other many types of manipulation tools.

Attachment handling and file renaming is superb in Bookends. The best part is: files are managed very transparently. You can explicitly put your files in a Dropbox folder without any hassle: while still they are attached to their references.

The Duplicate Finder tool embedded into Bookends is the best in the class. Jabref is great for finding duplicates. But it cannot reach the complexity and elegance of the duplicate finder in Bookends. You will never miss a duplicate with it.

If you are into Latex, it is also one of the most Bib friendly non-bib reference manager. It can assign unique Bibtex keys: and, the Format Manager can be manipulated to export in any of the Bibtex formats (Bibtex or BibLatex). You can also manipulate your format to export distinct (unique) bibtex fields. You don’t even need to export the reference to get your bibliography to work with Latex. You can make Bookends to work with your Latex file directly.

Bookends has  AppleScript dictionary. That again opens a world for who are into the scripting.

Integration with other super applications: I have never seen any reference manager as flexible as Bookends (I admit, Zotero could be even more flexible: I just don’t like that app; never tried it seriously.). For me, the fact that Bookends works well with Devonthink and Tinderbox is the best part of the story. Bookends is hand in glove with both Devonthink and Tinderbox. You can export and import in both directions; you can even directly sync them using some scripts (all the three are scriptable). Combining the three power tools is the nirvana for the knowledge worker. The world has never been greater.

The only feature I am still missing from Sente is the reading and annotating capabilities of Sente. The reading experience in Bookends is nowhere closer to Sente. Otherwise, as I am more using Bookends more, I am getting more surprises how a reference manager could be so rich and so capable. In an ideal world, BE would incorporate the annotation tools of Sente.

How to handle images in (Multi)Markdown text

Markdown is a language built for plain texts. The biggest weakness of plain texts is they can not hold images in them. To complement this fundamental weakness of plain texts, I have developed a system that works between Hazel, Dropbox and Keyboard maestro. The idea is simple. Create a permanent folder in Dropbox that you will maintain for the rest of your life. Whatever text you are writing in, whatever application you are using, always save and point to that specific folder. My folder lives in Dropbox. I call it simply Images. All of the pictures (images) that are part of my text notes are stored in this folder. Most of these images are made by snapshotting from the pdf files and online sources. For that matter, I made my mac to save all the screen-shoots  to automatically save in that folder. ok, let me write it like in steps:

  1. Make a folder in Dropbox: call it Images
  2. Make your mac to save the screen-shots in Images. Follow the steps in here to accomplish it

  3. Make a hazel rule to rename the screen-shots to a sequence of numbers (or some other pattern you like)

  4. Embed an applescript inside Hazel that will copy the file name of the image to clipboard

Look at the rule I made in Hazel: first only the rule;

ppic341

then with the script:

ppic342

 

Finally, use a Keyboard maestro macro to get the location of the image as well as the file name.

ppic343

That is it. What you need is just to snap the image from the PDF then, write “ppic” in your plaintext file. A link to your image will be inserted; and any markdown previewer (Marked for example) include the image in the preview.

Workflow with Sente, Devonthink, Scrivener using Hazel and Dropbox as glue: part 1

Since I want to follow up informations on the internet on a few applications that I am interested in,  I have setup  google alerts for tracking  blog posts and articles into my email inbox. Under my Devonthink tag, today I get this small visual workflow in a website called Pinterest.

workflow

I have never visited Pinterest before; but the visual illustration looks beautiful.  There is no explanation on how to build the workflow in the visual maps; but the illustration is elegant, something I have been planning to design. Since this is the  workflow  what I am already using it for the last year, it seems good idea to put how I put these different apps to work together to have the “virtually perfect” kind of workflow for my research (phd dissertation, I am starting soon). I have already written short articles in my previous posts on  some of the connections I made between these four majestic apps. In this extended post, I will demonstrate how I developed my workflow using Sente, Devonthink, Scrivener, nvALT, xMind, Sublime Text and ultimately Latex. Hazel, the Automator, Dropbox and Keyboard Maestro are the core glues of the workflow. I will spell out how each app works together with the other applications to great a coherent and elegant workflow.

Now, have a loot at a pictorial overview of the workflow we will have by the end of this series.

research workflow
Click for larger preview

In the next few series, I will try to explain each step of the workflow; how I develop the connections and how the tools chosen talk to each other to build a solid workflow. (I didn’t represent Hazel in this drawing mainly because the tasks of Hazel. The role of Hazel  will be clear by the end of the post)

Let me start from the app I use to gather resources for research: Sente.

# Sente

For me, the most important organization burden is lifted by Sente. Sente has four crucial features which are the life and soul of my workflow: Targeted browsing, File renaming,  QuickTags and Status. I will explain how I use each of these features to download, store, rename and organize my PDF files; then, how I will incorporate the organized PDF into the Devonthink hemisphere using Hazel as a glue.

I am not going to explain how to do all the importing and referencing process in Sente. That is left for the user manual; my task here is to show how to use what I call above the crucial features and getting things done.

Let me start from setting up the Sente library:

## File naming

The library is setup to store PDF attachments inside the Sente bundle. Setting up the library inside the bundle is very important for syncing the library to iPad via the Sente server.

The attachments will also be named as:

[First author Last name] [Year of publication] [Title of publication]

The file naming  is  important because I find some of the PDF files hard to read inside Sente; hence, I have to open them in acrobat. If you have the files properly named based on the author and title, getting the file is just a matter of second, specially if you are using Alfred.

## Targeted browsing

After you setup your library, the next step is to search files and download or import them into your sente library. There are two ways of getting your PDF files into Sente library; both of which support of-the cuff-setting up the reference of the file. The first method is directly downloading from the internet. Sente has this wonderful feature called targeted browsing. The main advantage of the targeted browsing is you have a choice of downloading bibliography information from a plethora of websites; you are not limited to Google Scholar or MedPub (the main weakness of Mendeley, by the way is absence of such a choice; it can download only from Scholar and medPub). Even if Google scholar is one of the greatest data sources on the internet, the data you retrieve from it are usually incomplete. Many people like MedPub. But, for my field, MedPub is irrelevant. Therefore, for articles, I havn’t found any better source than Google Scholar.  To get both the reference and the PDF file from Google scholar to your library, what you do is first import the reference information into your library by clicking the red button in   scholar website; and then, download the related PDF file. I am sure you know how to do this; I don’t need to explain it in depth.

The second approach is to have the PDF file in your disk; and then drag it or import it into your library. Sente will present you a window to add reference information to your PDF file. At this point, what I usually do is: highlight the title of the PDF file and take the title to my favorite search engines. For research articles, there is no better choice than Scholar.

As for books, thanks for their ISBN numbers, there are a lot of choices; WorldCat being one of the most popular. I used to retrieve the data for the books from WorldCat for a while.  But, after some time, I leaned that it sometimes confuses Affiliations metadata with Author. Therefore, I have been looking for alternative sources for retrieving bibliography information for books. Google Books is quite good; but Sente Targeted browsing seems to have some kind of difficulty to retrieve data from google books; takes longer time. Finally, , to my surprise,  I discovered the library of Stanford has the cleanest data; and Sente is very happy about it.

The template you develop to modify your targeted browsing in Sente is called Autolink Templates. Here is how I setup my Autolink Template:

Autolink Templates in Sente

 

## Quicktags

Quicktags: are the tools for organizing your research resources into groups. I have three major classes of Quicktags:

a) the Class: this is the class of quicktags that I assign to the PDF’s inherent classes. I assign these tags to classify the paper into the basic inherent classes of my field: Linguistics. Linguistics has many sub-branches if study; and sub-topics of research. Therefore, whenever I download a PDF paper, I assign these classes into the paper so that I can easily search and look at whenever I am looking at certain sub-topics. I for example use tags like Syntax, Semantics, Pragmatics, Phonology, plus sub-topics such as  VPs, PP, RC, DP under Syntax which what they are. I also have hierarchies of tags for the families of languages that I am interested in.

Afro-Asiatic [Semitic[Classical[Arabic, Hebrew]][Modern]] etc

b) the Project: this is a group of tags that I assign to papers on project basis. The projects are usually transient tasks that I plan, finish and move on to the next project. They could be part of the specific Class; they can also run across many classes. I assign these transient tags to the papers I am downloading, or on those which are already in my library. I search down my library, google scholar and may other source to combine all the resources to finish under one project using these Project tags.

c) the third group of tags is what I call meta tags. They are organizational tags. Whenever print a PDF file, I assign a tag xPrint; whenever I am reading the paper in Acrobat reader, I assign a tag called xAcrobat; or, finished reading the PDF and exporting the annotations of the PDF,  mark it xExport.

The Meta tags are supporters while Project tags are brothers of the Statuses, which I will explain in a moment.

## Statuses:

I basically use Statuses are meta-tags and to track the progress of projects. As you can see from the following snapshot, I have about 12 Statuses that I assign to my PDF files:

Statuses in Sente

– To be read nextis for example a status that I assign to a paper that I want to read just immediately after I finish reading the current paper.

  • Must read: is another status for a paper need to be read by hook or crook before I finish my PhD; a paper that I believe can offer a significant and profound insight to my research.
  • Repelling is on the negative side: a paper hard to read; or written in a bad language; the point is: I am dropping that paper; and might delete it from my library completely.

you get the idea

One fundamental problem with Docear

I have been watching the developments of Docear for a long time. I like the whole idea of research suite where you drop everything related with your research; and get things done without much distraction. There are a few applications which promise to give such a functionality both in the windows and Mac. Docear seems one of the most promising of these apps, specially given that the suite is an open source, any university/huge project pick it up and pushing it to perfection. The other advantage of docear is its integration of one of the best open source reference managers in existence. The idea of integrating mind-mapping into the research is also good. But, there is one serious issue that makes the application unusable to me for a long time; its poor capabilities to directly edit files. Since it is build on top of a mind-mapping application (Freeplane), editing even the most basic TXT files is currently impossible. If you want to write a text, you have to write it on the nodes of the maps and export it finally. That is the idea. But, personally I hate exporting and importing. If an app can index a folder from the explorer (Finder), what is the point of exporting and importing. Why couldn’t they give the chance for us to directly edit our files as they are stored in their folder, as Devonthink does, without the hassle of exporting and importing? To make the point clear: I have more than a thousand notes saved in TXT format in my dropbox folder. I can directly index that folder into Docear, which is great. But, I can not open these text files inside the application (I have to run another application to view them) which is unfortunately debilitating task for organizing the notes and finally write a draft.  The developer are probably thinking about short texts when they develop Docear. But, a serious writing is not about few pages; rather hundreds and thousands of pages of text. Write a hundreds of paragraphs, let alone pages on a node is a crazy idea in the first place. Even for a short text, mind-nodes are very ugly.  If docear has to be a true research suite, it has  get a good internal text editor. That is mandator.  Until this problem of text editing is solved, I find Docear useless to my workflow. I hope the developers will incorporate a good text editor into the app soon, and we will  enjoy the app.  

Syncing Scrivener with nvALT

nvALT is one great fast note writing app. After collecting your short notes for a long time into nvALT, now, it is time to use them in your writing. you have two choices:

importing: importing the notes saved in nvALT into scrivener. this is a simple method. The problems is, any of the changes you made in Scrivener will not be saved back into your nvALT notes. As a principle, I personally don’t like importing because all the notes I saved in plaintext in my dropbox folder will be eaten up by a proprietary file format…the reason I don’t like Evernote and confused about Tinderbox.

syncing: This is better method for working in Scrivener directly updates the notes in nvAlT and vise-versa. The best part, you are still maintaining your clean plaintext files in your dropbox folder. Suddenly you decide not to use Scrivener, well, everything is in you hard; open your texts in another editor. Therefore, my agenda is to show you a small trick to make the two best text editors work flawlessly together using Scrivener’s FolderSync feature.

  1. First, store your notes in nvALT, as a plain text file; in a folder in dropbox (finder). I call the folder nvALTnotes.
  2. Now, start up a new blank project in Scrivener. Before you write anything in it, go to File menu and Sync. Create a blank folder in Dropbox called Scrives and choose that folder in the sync dialogue.
  3. Now, open and look at the Scrives folder in dropbox. Scrivener has created two folders: Draft and Notes
  4. Delete the Notes folder
  5. Go to http://www.zibity.com/macdropany and install the small free app: Macdropany. What does the app do is create a symbolic link from your Finder to your dropbox. you can also do the symbolic link via Terminal if you are technically inclined. i am not. i go for the small app.
  6. Now, go to nvALTnotes folder–>right click–>Services–>Sync via MacDropAny
  7. Choose Dropbox from the list of cloud services
  8. Now, when the dialogue asks you to choose a folder in Dropbox, choose Scrives folder that you created above
  9. now, when the app prompts you to rename the folder, name is Notes (this is very important; otherwise, Scrivener will not sync)
  10. Finished.

Now, all your note files in nvALTnotes folder will be available inside Scrivener; and any changes you made inside Scrivener will be reflected in nvALT.

DEVONagent vs Google

DEVONagent is a specialized searching tool, designed to provide more accurate results than the regular search engines. I have been trying it for a few days now. So far, my experience is,  the search results are not  better or more accurate than regular Google search. I have tried a few queries using the Web Deepest field in DEVONagent. In many instances, the search results are even weaker than that of Google, in my view. Since the tools of DEVONagent seems to work by analyzing the keywords of web pages, they are generally deluded by some wiki pages which put a lot of keywords with little actual content. Google has a way of killing these kinds of sites which put too much key works (tags) without actual valid contents, as it receives feedback from the users’ experience.

Synchronize Devonthink and Scrivener

There is a small window of opportunity to make the two applications work together.The opportunity comes from the Sync feature of Scrivener.

Even if this feature oppens the oportunity, the fact that it has limited capability to sync files complicates the relationship bewteen the two applications.

The weakness of the Sync feature is; it doesn’t support multiple folders. This means that, your heirarchical organizations based of authors or topic you use inside Scrivener Binder ( Research or Draft folder) will not be available in the Sync folder. The sync folder will have only 2 folders; one for the Draft and another called Note for the rest of the files ( all the files inside the Research binder). All the items, in whatever heirarchy you put them insider the Research Binder will be put into a single flat folder. This makes things hard to identify which note belongs to which group (folder) of the Binder. In the current system of Sync, the feature is almost useless, specially if you have built complex system inside the Research Binder. Assume that you have collected your materials and grouped them under an author name inside Research Binder. Say, you have collected 50 notes, 5 PDFs and 10 webclipings under authorX. You also have 5 notes, 2 pdf and 5 clipings in AuthorY. Each of the authors have their own binder (folder) insider the Reacher binder. When you syn, all the notes, cliping and pds will mix insider a single folder called Note, insider finder. You can not distinguis which note belongs to which author then.
In ability to maintain the folder (binder) heirarchies is the main issue of the syn in Scrivener. This again makes it hard to index these notes into Devonthik because you don’t know which note belongs to which author (folder, group). So, to tackle this problem I have developed the followed the following steps. It looks complex, but, there is not better way, as far as I know.

1. Start a new project in Scriverner ( I am assuming you start from scrach, to make things simpler. If you already have a lot of folders, you have to figure out a way of dealing with it by yourself.)

2. Don’t import anything into Scrivern. First, put your files into different folders insider Finder.
Put all the author’s files inside a folder by the names of the authors.
AuthorX
AuthorY
AuthorY
3. Now, drag AuthorX only into the Scriverner’s Research binder
4. Now, Sync the library of Scrivener ( got to File > Sync). When you sync, Scrivern will ask you to choose a folder. Creat a new folder, probably inside Dropbox. I call the new folder ScrivenerX.
5. Now, go to the ScrivenerX and have a look at the files. Scrivener has created two folders during the sync; Draft and Note folders. Draft contains all the items in the Draft binder while Note contains all the items inside the Research binder. Now, you must be able to see all the notes of AuthorX inside the Note folder.
At this point, you can tag them all by the name of the AuthorX. This makes them searchable even in Spotlight for other times. (I have Tagger in the toolbar of my finder. i use it to tag all the files at the same time)

6. Index all the files insider the Note folder into Devonthink (hold Alt+CMD and drag them into a group called AuthorX inside Devonthink). This step is very necessary. If you import and Sync AuthorY before you indexing the files of AutorX, you will not again able to identify the files inside Devonthink.

Now, you are finished syn and tagging Author X. Repeat the steps for other authors: (drag the folder Y into Scrivern > sync Scriverner >tag the newly imported files inside Note folder >Drag them to Devonthink)

Here you go, , your Scrivener and Devonthink are in Sync finally. Anything you edited in Devonthink will be appear in Scrivener; and things you edit in Scriverner will appear in Devonthink, as far as you don’t forget to Sync inside Scrivener.