Mirroring ‘Oxford Scholarship Online’ on your mac using Bookends

 

Oxford publications started up a sort of revolution on how printed books are accessed online.

Most other publishers put the exact same book, in a single PDF or Epub format, online for sale. Oxford makes an important change that seems to start a revolution on how we consume books online. In Oxford’s system, each of the chapters of the books stand by themselves as published articles. Each of the chapters of the collection gets their own DOI that they can be read, downloaded or referred independently of the main book.

Look at this video how it works.

Now, we are in a position to have the cake and eat it at the same time. The old conflicts on whether to keep books broke into their chapters and sections (as such a system facilitates searching and discovery on each of the chapters), or, keep it together to be able to see the work as a whole immediately disappears. We have both the whole and the individuals at the same time.

So, I have been wondering on how to mirror that system in Bookends. I experimented on how each of these independent chapters could be managed. Bookends has a mechanism to manage Chapters separately; while still linked to the main book.  Bookends is such a well-thought application: each of the chapters are able to live by themselves just exactly like they are presented in Oxford Scholarship Online. I am so impressed. I haven’t seen any other offline reference manager that is able to manage chapters like Bookends (Oxford).

The more I use each of the features, the more appreciate how well-designed Bookends is.

For me, I used to split PDF books into specific page ranges (page 1-50; 51-100 etc) just to keep the large pdf books easier for my searching tools like Foxtrot. I have already explained the system before.

Now, with Oxford Scholarship online and Bookends, I don’t need to split. I simply download each of the chapters and attach them as chapters into a single reference entry in Bookends. The pdf files of each of the chapters are attached to their corresponding entries. It is much better system because I won’t have duplication of information. I only hope that all other publishers offer books in the same way to the Oxford Scholarship online.

 

Advertisements

Comparing the 3 best pdf readers in Mac: Skim vs Highlights vs PDF expert

PDF files are the life of the academic. All information come with them. Of the time I spend reading, more than 95% of it goes by Pdf files. For that end, a good pdf reading application would be very important: much more important than any other application I use.

I have been looking for different tools for reading Pdf files.

Every one of them have strengths and weakness:

1. Skim

Skim: designed for the academic community: free and open source.

Strengths:

  • The ranked search is amazing.  of the  3, Skim has the best searching capabilities. PDF  expert comes next.
  • The annotations are more powerful and flexible: the anchored note is specially a wonderful tool. You can literally draft your next book using the anchored note. Reading triggers ideas; ideas breed ideas. Ideas cannot come out of the blue: they emerge during the reading. The best part of the anchored notes that you can give Titles to the notes. You can manipulate them so that the exported note will be much cooler.

I usually put ## on the title of the anchored note so that the title will come out as a true title when I exported the annotation using Markdown format.

  • The keep on top feature is very useful to compare ideas: any of the annotations can be kept on top.
  • Supports scripting
  • Export templates: you can modify these templates to your need. This is extremely useful.

But, there are some weaknesses with this all strength.

  • The non-standard format: if you want to read or see the annotations of the Skim, you have to export it. You cannot just open the pdf and keep on annotating. This is a deal breaker. Really. I am tied up to local system; this is like a prison. I like PDF expert for it adheres to the standard PDF specifications. I think it is the best mac reader with the standard formats next to Adobe’s own products.  Mac OS has this weird system comes by the name **PDFKit**: it gets broken, keep on screwing us all the time. Corporate greed seems the reason why we are suffering. Why doesn’t Apple adhere to the standard Adobe specifications? This same crap Kit also seems the  reason that saving pdf files in Preview and the rest of Mac local applications bulges up the size of the pdf.
  • The separate .SKIM file is a pain in the ass. It gets lost. If you export and import the pdf, all the annotations get duplicated. It is whole mess.

If Skim follows the standard PDF specifications; writing the annotations directly to the PDF itself, I would never look around.

2. Highlights

Strength:

  •  It follows the standard annotation system: annotations made in Highlights can be viewed and edited in other editors (both in the mac and windows)
  • The annotations are powerful. The annotation panel could be wide: therefore,  a long text can be directly inserted. Even if it is not as convenient as the anchored notes in Skim, the panel is generally convenient to drop longish texts.
  • Works great with other applications: like Devonthink, Bookends, and Evernote. This is one of its best features
  • the exported notes are in Markdown format: this can be taken as strength and weakness: depending on your interests.
  • Splitting annotations into distinct notes. This is the most interesting feature, for me, because I can keep single ideas as separate notes. I  have been using Sente annotations for this purpose.

Weaknesses

  • general clunkiness: the app contains a lot of bugs
  • the Splitting feature is not well worked out. I would have bought this app if I were able to assign titles to each of the annotations. The Titles are very useful for summarizing the concepts of each of the singular annotations. This is the most debilitating problem I have with Highlights. The spliced notes have no meaning: not life because they are not customized by titles or tags.

3. PDF expert

PDF expert is very fast and fluid application. I use it everyday. The developers are generally very responsible and fast guys. The code they write is amazing. The programmer talent in Readdle  tend to be very high. I participated in their beta versions for a long time now. I can tell you, their betas are more matured and reliable than the final releases that Apple sends out. There are some small details: specially its speed, which makes this app worth trying. I like it so much. I is the first app I open in the morning. The best part of the app is that it follows the standard Adobe system. The annotations made in PDF expert are visible on any other pdf reader. That is why I use it as my default reader.

Strengths:

  • I also like the new searching tool. It searches all the open files.
  • it automatically detects the true pages numbers of the pdf
  • blazing fast
  • follows the standard Acrobat annotation format
  • The annotation tools are generally ok

Unfortunately, PDF expert is also  the least creative of the 3 apps I am trying. The features it contain are most already in acrobat or other pdf readers.  The export features are very weak: even terrible. I tried to export in the Markdown format. It doesn’t permit me to customize on what types of text I want to export. Generally, the exported text turn out to be  vary bad containing unwanted stuff (like Date, author…).  Even if there is wondrous programming talent,  the direction they are taking are mundane and non-creative.There is barely a new feature in this reader that other readers, like PDFPen, Acrobat etc, do not have.   They don’t understand the areas of need. The annotation tools could  be better. The developers of Highlights have truly understood the needs of the scientific community.  It is only the implementation that is lacking in the latter.

My take:

After trying it on a couple of times, I have given up with highlights.

I am now using Pdf expert and Skim. I use Pdf expert for fast reading. When I have to just scan and take a few points, I open my file with it. I highlight a bit; clip a few lines to Curiota and close it down.

When I have to read a book or an article from the beginning to the end, for intensive reading, no reader offers the comfort that Skim offers. The exports are also much robust. Therefore, for in-depth reading, I am relying on Skim.

By the way, there is an other candidate that could offer a similar comfort for reading: the Marginnote. It seems to have some great annotation tools similar to anchored note: even better, mind mapping within the reader. I tried it for a couple of minutes. But, I dropped the app immediately because the annotations are in proprietary format: they will be a big lock down. While the annotation summary can be exported, the annotation and the annotated pdf are divorced forever. I am skeptic of apps that highly rely with proprietary file system.

Where Bookends rocks

Before, I wrote a few complains (https://dellu.wordpress.com/2016/03/28/where-bookends-sucks/) I have on Bookends as replacement for Sente. It has been almost 9 months since that post.

Now, I am updating my experience with the Bookends.

The good news: the latest version of Bookends (12.7.7) has solved one of the major complains I had on the application. That is: it can now extract references from a plethora of sources that the source of extraction is no more a problem.

In addition, I learned in the course of the previous months that the right method of finding references of PDF books is to use the ISBN associated with the books. Once you write the ISBN of the book in the ISBN field, Bookends happily can download all the reference data from Amazon using its Autofill feature. The Autofill feature is a super-feature. One day,  I put about 400 PDF articles collected over the years into Bookends watch folder; all of them have their DOIs printed in the first pages of the articles. It was just magic to see all the PDF files find their references filled automatically in a couple of minutes.

I also leaned that Bookends is much potent application than Sente when it comes to managing references. The tools embedded into it are unbelievably rich. It has organizational tools like: smart folders which support REGEX,  static folders,  labelling (color coding). It has a fully configurable Format Manager which enables the end user to import or export; rename the references in endless ways. The Format Manager in Bookends is the greatest feature I have ever seen in any reference manager. It is a dream come true. It is a liberating feature. I cannot say enough about it.

Bookends has other tools under Global Change. Finder replace that works across the whole library; batch changing of reference types, batch inserting data..and, other many types of manipulation tools.

Attachment handling and file renaming is superb in Bookends. The best part is: files are managed very transparently. You can explicitly put your files in a Dropbox folder without any hassle: while still they are attached to their references.

The Duplicate Finder tool embedded into Bookends is the best in the class. Jabref is great for finding duplicates. But it cannot reach the complexity and elegance of the duplicate finder in Bookends. You will never miss a duplicate with it.

If you are into Latex, it is also one of the most Bib friendly non-bib reference manager. It can assign unique Bibtex keys: and, the Format Manager can be manipulated to export in any of the Bibtex formats (Bibtex or BibLatex). You can also manipulate your format to export distinct (unique) bibtex fields. You don’t even need to export the reference to get your bibliography to work with Latex. You can make Bookends to work with your Latex file directly.

Bookends has  AppleScript dictionary. That again opens a world for who are into the scripting.

Integration with other super applications: I have never seen any reference manager as flexible as Bookends (I admit, Zotero could be even more flexible: I just don’t like that app; never tried it seriously.). For me, the fact that Bookends works well with Devonthink and Tinderbox is the best part of the story. Bookends is hand in glove with both Devonthink and Tinderbox. You can export and import in both directions; you can even directly sync them using some scripts (all the three are scriptable). Combining the three power tools is the nirvana for the knowledge worker. The world has never been greater.

The only feature I am still missing from Sente is the reading and annotating capabilities of Sente. The reading experience in Bookends is nowhere closer to Sente. Otherwise, as I am more using Bookends more, I am getting more surprises how a reference manager could be so rich and so capable. In an ideal world, BE would incorporate the annotation tools of Sente.

Mendeley vs Citavi vs Qiqq (also Sente and Bookends)

I haven’t used my Windows machine for a while now. I was then curious how the reference managers progressed in these periods. I was specially curious about Mendeley because I struggled with that application for some time then.

So, here is my observation: Mendeley stayed the same for the last couple of years. There is no real development; nor any change of any relevant sort since I knew the application. All the icons, the settings, the menus; the features: I see no changes. It is as clumsy as used to be; in many areas. One of the properties that Mendeley sucks at is how reference is downloaded from the internet. It attempts to use Google scholar; combining with its metadata extraction too. What it does is: it attempts to detect some DOI or other identifier to the PDF in the first few pages; and then, use that information to download reference information from Google scholars. For me, the result is a total debacle. It has always been a debacle. Mendeley can detect the papers only less than 5% of the times; as my PDFs don’t typically have metadata information; nor are they always published articles. Many of them are books; or drafts of books, and earlier versions of published articles I received from friends.

My favorite feature of Mendeley, which had been, still is: the BibTex sync feature. I have to admit, I have been tempted to live with Mendeley because of that feature. But, heck, if you have wicked reference data, what is the point of syncing it to Bib file. You will have incomplete citations ultimately. You will be embarrassed in front of colleagues when you realize that your references are incomplete after you sent out the paper. Because of the importance of the feature, I will focus on this feature in comparing the reference managers.

Citavis is not very far better than Mendeley when it comes to reference extraction from the internet. It can even be worse. I was able to download references from the internet only if the book has ISBN numbers or the article has DOI number. Otherwise, manual insertion is the only way I am left with. Look at this tutorial to learn how the process is clumsy in this application: https://youtu.be/MyaW9q_464w?list=PLkLfx87WKrTZTfifTvttqgKwzyQSTDYY5

In Citativ, when you read a PDF file, you can highlight or quote a certain text: comment on it; or give a short title to the comment and the quote. I totally love the idea of giving a short title to the quotation I make from a PDF reading. This feature is also available in Sente. The idea is: you quote a certain sentence or paragraph from the PDF; then, give a title which summarizes the core point of the quote and tag it if you want to. These quotes serve as a short summary of the article. The titles are your reminds of the core points of the quote. It is like summarizing the summary. Very neat approach to reading articles. The neat part in Sente is each of these short quotes could be exported as a separate note file. That means, if you have 20 quotations from the article, you will have 20 short notes: titled appropriately in a folder in finder. The problem with Citavi is each of the quotations are not exportable to separate notes. They can be exported as single file only. That means, it is not any better than reading and annotating a PDF in Acrobat Reader or other PDF readers (PDF exchange; or Foxit in windows: PDF expert in Mac)  and exporting a summary.

Qiqq is very different. Its way of extracting references from Google scholars is comparable to Sente. You click the PDF; click BibTex sniffer: you will be given Google scholars to pick the references. If Qiqq failed to detect the title of the PDF correctly, you can manually select the title. Qiqq immediately populates the scholar search with the selected text.

I have given the following book for all the three reference mangers: It was only Qiqq which correctly imported the full reference information.

gq4omd

I think the reference downloaded in Qiqq is much better than Sente. Sente has an advantage of downloading from multiple sources like WorldCat; Stanford, British libraries…very good results in some sources, weaker results in others. I used to get the most complete reference data from Stanford library website. But Sente sometimes fails to download the Publisher Field from many sources. Bookends can pick from Google Scholar, Justor and two other sources. But the process  of downloading a reference data (called Autocomplete in Bookends; targeted browsing in Sente; BibTex sniffing in Qiqq) is most elegant in Qiqq and Sente.

 

The other interesting feature of Qiqq is the brainstorming feature: absolutely brilliant tools to play with your references. It can also be used to track the positions one author took over time, how his/her ideas change in the long run. it can also be used to study the history of ideas: where a certain phrase appeared first; then, how other authors reflected that phrase in their publications. Look at these tutorials to see how the Brainstorming works in Qiqq:

 

My ratting of these reference manager’s capability of downloading references from the internet:

  1. Mendeley = 4/10
  2. Zotero= 3/10
  3. Citavi = 2/10
  4. Bookends = 7/10
  5. Sente = 9/10
  6. Qiqq = 8/10

Why is Sente higher in this ranking?

Because it offers much better choice than Qiqq on the sources. Qiqq does it elegantly on Google scholar; but, it cannot download from other sources which potentially offer more complete reference data.

Conclusion: if I ever have to move to Windows, I will definitely use Qiqq (in combination with OneNote or ConnectedText).

Why you need to split your big PDF books

I have one secret tool that I bust all my class mates when it comes to digging down the nitty-gritties of small pieces of information. 

When we discuss some issue with professors or classmates, sometimes we come up with some wild ideas. We ponder about it; ask if anybody else has thought that before us. What they usually do is google.  I also sometimes google the ideas if anybody else thought them  before us (me). But, the fact of the matter is, google has a lot of noise out there with the same keywords but has little to offer the very specific information I am looking for. 

That is where a internal database comes to rescue. I collect as many books and article into my disk so that I can dig them whenever i want to find out specific ideas. The concept is known by “text mining” in a different camp of linguistics.

 Right now, I have over 2000 books and articles in my disk all of which deal with Theoretical Linguistics. 

If you have a collection of books and articles like me, and tried search a specific phrase into it; using Alfred, Spotlight or Devonthink, you will immediately learn that the biggest book always comes on top regardless of the quality of the material in it. The reason behind it is the word count. The larger the book, the more likely that it contains the queried word multiple times. If you collection specially contains gigantic Encyclopedia books, there is not chance that the short article comes out on top of your search result however relevant the article could be. 

Therefore, to make each small article as competent as any other material; and that your search tools could pick the small articles whenever they are relevant, you need to split the books into article sizes. 

I have experimented with different tools of splitting my books; beginning from Apple’s own Automator to a number of python and Shell scripts. Most of them work by bursting  the book by pages.

Bursting a book into single pages could be feasible when you have less than 1000 books. As you books grow, the bursting creates too many files to manage. In addition, the single pages won’t contain enough material to read within  the search result (FoxTrot for me). That is where splitting in 10-15 (article size) rage becomes crucial. 

Right now, I have shell script that breaks down my books in 10 pages ranges, a script that I adopt from a South African guy (I forget his name; I met him in Acadamic.edu). 

Any book or article I add to Sente library directly gets copied to another folder (using Hazel ) and gets splitted into article size page. All the pages finally move to another folder for ultimate archival; where my searching tools such as Devonthink and Foxtrot index. 

I will come back  to the  the full workflow and the scripts I use to achieve the task in another post. 

Workflow with Sente, Devonthink, Scrivener using Hazel and Dropbox as glue: part 1

Since I want to follow up informations on the internet on a few applications that I am interested in,  I have setup  google alerts for tracking  blog posts and articles into my email inbox. Under my Devonthink tag, today I get this small visual workflow in a website called Pinterest.

workflow

I have never visited Pinterest before; but the visual illustration looks beautiful.  There is no explanation on how to build the workflow in the visual maps; but the illustration is elegant, something I have been planning to design. Since this is the  workflow  what I am already using it for the last year, it seems good idea to put how I put these different apps to work together to have the “virtually perfect” kind of workflow for my research (phd dissertation, I am starting soon). I have already written short articles in my previous posts on  some of the connections I made between these four majestic apps. In this extended post, I will demonstrate how I developed my workflow using Sente, Devonthink, Scrivener, nvALT, xMind, Sublime Text and ultimately Latex. Hazel, the Automator, Dropbox and Keyboard Maestro are the core glues of the workflow. I will spell out how each app works together with the other applications to great a coherent and elegant workflow.

Now, have a loot at a pictorial overview of the workflow we will have by the end of this series.

research workflow
Click for larger preview

In the next few series, I will try to explain each step of the workflow; how I develop the connections and how the tools chosen talk to each other to build a solid workflow. (I didn’t represent Hazel in this drawing mainly because the tasks of Hazel. The role of Hazel  will be clear by the end of the post)

Let me start from the app I use to gather resources for research: Sente.

# Sente

For me, the most important organization burden is lifted by Sente. Sente has four crucial features which are the life and soul of my workflow: Targeted browsing, File renaming,  QuickTags and Status. I will explain how I use each of these features to download, store, rename and organize my PDF files; then, how I will incorporate the organized PDF into the Devonthink hemisphere using Hazel as a glue.

I am not going to explain how to do all the importing and referencing process in Sente. That is left for the user manual; my task here is to show how to use what I call above the crucial features and getting things done.

Let me start from setting up the Sente library:

## File naming

The library is setup to store PDF attachments inside the Sente bundle. Setting up the library inside the bundle is very important for syncing the library to iPad via the Sente server.

The attachments will also be named as:

[First author Last name] [Year of publication] [Title of publication]

The file naming  is  important because I find some of the PDF files hard to read inside Sente; hence, I have to open them in acrobat. If you have the files properly named based on the author and title, getting the file is just a matter of second, specially if you are using Alfred.

## Targeted browsing

After you setup your library, the next step is to search files and download or import them into your sente library. There are two ways of getting your PDF files into Sente library; both of which support of-the cuff-setting up the reference of the file. The first method is directly downloading from the internet. Sente has this wonderful feature called targeted browsing. The main advantage of the targeted browsing is you have a choice of downloading bibliography information from a plethora of websites; you are not limited to Google Scholar or MedPub (the main weakness of Mendeley, by the way is absence of such a choice; it can download only from Scholar and medPub). Even if Google scholar is one of the greatest data sources on the internet, the data you retrieve from it are usually incomplete. Many people like MedPub. But, for my field, MedPub is irrelevant. Therefore, for articles, I havn’t found any better source than Google Scholar.  To get both the reference and the PDF file from Google scholar to your library, what you do is first import the reference information into your library by clicking the red button in   scholar website; and then, download the related PDF file. I am sure you know how to do this; I don’t need to explain it in depth.

The second approach is to have the PDF file in your disk; and then drag it or import it into your library. Sente will present you a window to add reference information to your PDF file. At this point, what I usually do is: highlight the title of the PDF file and take the title to my favorite search engines. For research articles, there is no better choice than Scholar.

As for books, thanks for their ISBN numbers, there are a lot of choices; WorldCat being one of the most popular. I used to retrieve the data for the books from WorldCat for a while.  But, after some time, I leaned that it sometimes confuses Affiliations metadata with Author. Therefore, I have been looking for alternative sources for retrieving bibliography information for books. Google Books is quite good; but Sente Targeted browsing seems to have some kind of difficulty to retrieve data from google books; takes longer time. Finally, , to my surprise,  I discovered the library of Stanford has the cleanest data; and Sente is very happy about it.

The template you develop to modify your targeted browsing in Sente is called Autolink Templates. Here is how I setup my Autolink Template:

Autolink Templates in Sente

 

## Quicktags

Quicktags: are the tools for organizing your research resources into groups. I have three major classes of Quicktags:

a) the Class: this is the class of quicktags that I assign to the PDF’s inherent classes. I assign these tags to classify the paper into the basic inherent classes of my field: Linguistics. Linguistics has many sub-branches if study; and sub-topics of research. Therefore, whenever I download a PDF paper, I assign these classes into the paper so that I can easily search and look at whenever I am looking at certain sub-topics. I for example use tags like Syntax, Semantics, Pragmatics, Phonology, plus sub-topics such as  VPs, PP, RC, DP under Syntax which what they are. I also have hierarchies of tags for the families of languages that I am interested in.

Afro-Asiatic [Semitic[Classical[Arabic, Hebrew]][Modern]] etc

b) the Project: this is a group of tags that I assign to papers on project basis. The projects are usually transient tasks that I plan, finish and move on to the next project. They could be part of the specific Class; they can also run across many classes. I assign these transient tags to the papers I am downloading, or on those which are already in my library. I search down my library, google scholar and may other source to combine all the resources to finish under one project using these Project tags.

c) the third group of tags is what I call meta tags. They are organizational tags. Whenever print a PDF file, I assign a tag xPrint; whenever I am reading the paper in Acrobat reader, I assign a tag called xAcrobat; or, finished reading the PDF and exporting the annotations of the PDF,  mark it xExport.

The Meta tags are supporters while Project tags are brothers of the Statuses, which I will explain in a moment.

## Statuses:

I basically use Statuses are meta-tags and to track the progress of projects. As you can see from the following snapshot, I have about 12 Statuses that I assign to my PDF files:

Statuses in Sente

– To be read nextis for example a status that I assign to a paper that I want to read just immediately after I finish reading the current paper.

  • Must read: is another status for a paper need to be read by hook or crook before I finish my PhD; a paper that I believe can offer a significant and profound insight to my research.
  • Repelling is on the negative side: a paper hard to read; or written in a bad language; the point is: I am dropping that paper; and might delete it from my library completely.

you get the idea

One fundamental problem with Docear

I have been watching the developments of Docear for a long time. I like the whole idea of research suite where you drop everything related with your research; and get things done without much distraction. There are a few applications which promise to give such a functionality both in the windows and Mac. Docear seems one of the most promising of these apps, specially given that the suite is an open source, any university/huge project pick it up and pushing it to perfection. The other advantage of docear is its integration of one of the best open source reference managers in existence. The idea of integrating mind-mapping into the research is also good. But, there is one serious issue that makes the application unusable to me for a long time; its poor capabilities to directly edit files. Since it is build on top of a mind-mapping application (Freeplane), editing even the most basic TXT files is currently impossible. If you want to write a text, you have to write it on the nodes of the maps and export it finally. That is the idea. But, personally I hate exporting and importing. If an app can index a folder from the explorer (Finder), what is the point of exporting and importing. Why couldn’t they give the chance for us to directly edit our files as they are stored in their folder, as Devonthink does, without the hassle of exporting and importing? To make the point clear: I have more than a thousand notes saved in TXT format in my dropbox folder. I can directly index that folder into Docear, which is great. But, I can not open these text files inside the application (I have to run another application to view them) which is unfortunately debilitating task for organizing the notes and finally write a draft.  The developer are probably thinking about short texts when they develop Docear. But, a serious writing is not about few pages; rather hundreds and thousands of pages of text. Write a hundreds of paragraphs, let alone pages on a node is a crazy idea in the first place. Even for a short text, mind-nodes are very ugly.  If docear has to be a true research suite, it has  get a good internal text editor. That is mandator.  Until this problem of text editing is solved, I find Docear useless to my workflow. I hope the developers will incorporate a good text editor into the app soon, and we will  enjoy the app.