Head Tilting Detection - Automatic Page Turner for Pianists

Head Tilting Detection (or Automatic Page Turner for Pianists, still undecided about the name ūüėõ ) is a simple software that emits a Page Down/Up keypress when the user is tilting the head.Photo-3

Since I started learning piano, more than 10 years ago, I had the problem of turning pages. Turning pages is one of the most annoying thing for a pianist: it forces you to waste seconds, interrupt the flow of music, and it affects the way we learn the pieces (e.g. by making the connection between pages really poor, since we usually stop from a page to the next). Several alternatives exist for  turning pages automatically, but they are clumsy and inefficient. Recently I thought about applying my programming knowledge to this purpose.

As more and more pianists are switching from paper to digital scores, it is possible to use a machine learning approach. I designed a simple software that detects when you are tilting your head right/left, and "scroll" down the page accordingly by simulating a page down/up keypress, with in most software will scroll the window down/up - in Adobe PDF, if you "fill one full-page to window", you will turn a whole page (to next/previous one).

Update 28/03/2016: New version for Linux released. You can now rotate/flip your webcam viewer. The sound now works on Windows machine.

WINDOWS 32/64bit (tested on Windows 7, Windows 8, Windows 8.1)

Download Head Tilting Detection (no setup required) - for Windows

LINUX (tested on Ubuntu 15.10)

Download Head Tilting Detection (no setup required) - for Linux

open folder in terminal and type sudo ./HeadTiltingDetection.sh. This bash file will download and install xdotool if not installed already) and the software will be executed.

Instructions

Open the software, wait for the your face to be detected (a green rectangle around your face should appear), then wait for your eyes to be detected (red squares).
At this point select the application you want to send the keypress (for example, a pdf file).
When you tilt the head on one side, the corresponding arrow key signal will be emitted. A green circle should appear on the corresponding side of the camera view, and you should hear a specific sound for the direction you are tilting your head (you can disable the sound).

Adjust the threshold from the slider, disable the sound, or pause the detection from the user interface.

Key points

  • The software is designed for pianists, so with more or less constant level of light, by taking into account slow/random movement, and by considering that pages are usually not turned so often. Plus, instead of just "turning" your page, it will scroll down on the page, with in most of the cases is what you want.
  • For now, the software does not work if you wear glasses. I will maybe work on this in the future.
  • The software uses a Haar feature-based cascade classifiers, with template matching for tracking the eyes, and several heuristics to increase accuracy. A cubic function of the two eyes' degree is used to emit the keypress. The threshold for the keypress can be adjusted.
  • The software has been tested on Windows 7, 8, 8.1. For now there is only a Window version, but I may develop it for other platform in the future. (28/03/16) Linux: Ubuntu version released

Future work

The software is not perfect. Unfortunately, I do not have many hours to dedicate to it. I plan to keep work on it, but not consistently. I'll probably add to this post any minor updates, and create new posts only if there is a major improvement. If you want to collaborate with me for this project, feel free to contact me!

The software uses OpenCV (for the computer vision part) and Qt. It was very nice working with Qt again after so long, and discovering OpenCV was also very interesting. I have a very good opinion about both in terms of usability and capability. Managing to merge both systems was an excellent experience for me.

Copy highlighted text into comments from a PDF file

One of the nice features of Acrobat is that you can highlight text and then export only the highlighted part into a different document.  However, in order to do that, the user has to remember to tick the option "Copy selected text into Highlight, Cross-Out, and Underline comment pop ups" in Edit - Preferences - Commenting. Unfortunately, this setting is not on by default and only available  in Acrobat (8,9 and X) but not in Acrobat Reader, as said here.

It can happens that we highlight a really big document in order to export the highlighted parts, and then we remember that the "Copy selected text..." was actually off! Our highlighted parts won't be commented, meaning that we cannot export them (actually, we *can* export them, but they will be just empty boxes). There is no way to retroactively  copy all the highlighted part into comment from the graphic interface.

screensh2

However, it is possible to solve this problems by using some code. This solution will also work if you have Acrobat Reader!

There is already a software online that does that, on this website, but the price is really high (from $40 to $75!), so I developed my own script and I am going to put it out for free:

HOW TO RETROACTIVELY COPY HIGHLIGHTED TEXT INTO COMMENTS IN ACROBAT PDF

1. OPEN JAVASCRIPT EDITOR. First of all open the document with the highlighted text, and press ctrl+j. In Acrobat Pro, this should open the JavaScript editor window. If it works, just go to point 2, otherwise keep reading.

It ctrl+j doesn't do anything, than you don't have Acrobat Pro, but don't worry, you can still open the JavaScript editor. Go on this website and download the Reader JavaScript Console Window. Download the file anywhere, and follow the instruction in the ReadMe file. Once done that, you should see a new voice in the PDF menu, "Extension", and clicking on "Debugger" should open the JavaScript editor window. Done!

2. COPY THE SCRIPT. Delete any text on the bottom window, and copy and paste this code:

var annots = this.getAnnots({nSortBy: ANSB_Page});
console.println("nAnnot Report for document: " + this.documentFileName);
if ( annots != null ) {
console.println("Number of Annotations: " + annots.length);
var annotList=[];
for (var i = 0; i < annots.length;i++) {
        
        var annotTxt="";
        while (annots[i].type!="Highlight") {
            annotTxt="****";
            i=i+1;
        }
        if (i>=annots.length) {
             break;
        }
      
        pageNum=annots[i].page;
        var quadAN=annots[i].quads.toString();
        var qaAN=quadAN.split(",");
        for (var ii=0; ii<qaAN.length; ii++) {
            qaAN[ii]=parseFloat(qaAN[ii]);
        }
        for (var w=0; w<getPageNumWords(pageNum); w++) {
            var quadWD=getPageNthWordQuads(pageNum,w).toString();
                        //console.println("QWD TYPE: " + typeof quadWD + ": " + quadWD)
            var qaWD=quadWD.split(",");
            for (var ii=0; ii<qaWD.length; ii++) {
                qaWD[ii]=parseFloat(qaWD[ii]);
            }
            
            var nlines=qaAN.length/8; counter=0;
            for (var nn=0; nn<nlines; nn++) {
                if (qaWD[0]>=qaAN[counter+0]-4.5 &&
                    qaWD[1]<=qaAN[counter+1]+0.5 &&
                    qaWD[2]<=qaAN[counter+2]+4.5 &&
                    qaWD[3]<=qaAN[counter+3]+0.5 &&
                    qaWD[4]>=qaAN[counter+4]-4.5 &&
                    qaWD[5]>=qaAN[counter+5]-0.5 &&
                    qaWD[6]<=qaAN[counter+6]+4.5 &&
                    qaWD[7]>=qaAN[counter+7]-0.5) {
                    annotTxt=annotTxt+" "+getPageNthWord(pageNum,w);
                }
            counter=counter+8;
            }
        counter=0;
            
        }
        //UNcomment one line below if you want to show information about the annotations
        //annotTxt="ANNOT N." + i + " PAGE NUM: " + (pageNum+1) + " : " + annotTxt;
        annots[i].contents=annotTxt;
        annotList[annotList.length]=annotTxt;
        //UNcomment the line below if you want to print on the screen the annotations
                //console.println(annotTxt)

}

} else
console.println(" No annotations in this document.");

console.println("DONE!")

Select all the code (ctrl+a) and execute it (ctrl+enter). You have done! If everything has been done right, the highlighted part will now contain comments, as shown in figure:

screensh3

DONE.

Now you can be easily export the highlighted annotations on a PDF or word document. You can use the Acrobat option of exporting contents, as explained in this video. However, with the code we have now, you don't really need to do that: you can just uncomment (delete the double slash, //)  line 53 and execute the script again. This line will print on the screen the found highlighted words. If you want to also show some information about the highlighted part (just the annotation number and page) uncomment line 49. Execute the code and copy-paste the output anywhere. However, if you want, you can still use a more classic way, explained for example

The process of generating the comments could take some time, so be patient, even if the edito and the PDF will look freezed! If you really believe that the editor got stuck, try to press ctrl+shift+esc to interrupt the execution. If you have a lot of highlighted annotations (let's say, more than a hundres), it is a lot better to execute only chunk of text. To do that, change line 6 . For example, change it first to

for (var i = 0; i < 100;i++)

then to

for (var i = 100; i < 200;i++)

But always make sure that the second index will be not higher that the tot number of annotations.

PERSONAL NOTES:

I actually needed to export highlighted text because I was building a software to create the keywords index of a book, and the keywords were actually the highlighted part of the text itself. Since I never never used JavaScript before, this turned out to be quite a challenge. The way to get the text is extremely convoluted, and it' s really incredible that the Acrobat API doesn't have an easy way to do that. Briefly, I take the coordinate of the boxes describing the position of the annotations (the highlighted text). If the annotation goes through multiple lines, I'll get multiple coordinates and split them in chunk of 8. Fortunately, the annotation object also give me the page it is present. For each annotation, I will go through all the words within that page, take the coordinate of the words, and check if they are within the coordinates of the annotations. Strangely, the y coordinate starts from the bottom of the screen, instead the from above, as common in programming languages. This method will work also with words that are separated across two line.

I also noticed that, for some reason, if the word is close to some punctuation, the coordinates will mistakenly report an higher value (for example, for the word -hi- (within the dashes) the word coordinates will start and end when the dashes start and end (even if the actual word read if only hi). Therefore, if the only highlighted part is hi, the word will not be recognized, because the coordinates of the word will lie outside the coordinates of the highlighted box. To solve this problem, I just slightly increased the coordinates for the boxes (the +-4.5 or +-0.5 in the code), and this solved the problem completely.

Anyway, my first experience with JavaScript is totally positive. The language is intuitive and extremely similar to Java and C++, which I am more confident with. The real problem was the Acrobat API, which seems to behave in an unpredictable way.

-----------------------------

A big thanks to Francisco Morales which, with this post, pointed me to the right direction to develop this script.

If you appreciate my effort, please leave a comment!
If you have any problem with the script, don't be afraid to ask! ūüôā

Reference Creator

In my first post I will talk about the project that led me think about opening a blog. This hasn’t been the most difficult thing I have worked on, but the one I put more effort to make it accessible to everyone.  I am talking about my Reference Creator, which can be downloaded here.

Screenshot
This software is design to be helpful for people writing articles. In my experience, people tend to write scientific article putting the in-text citation, without worrying for long time about the extremely boring task of organizing the reference. If you are not using Latex or some particular text editor, creating the reference can be an incredibly tedious task. And that’s when my software comes in handy! It will just take your in-text citations and create a reference list for you!

Let‚Äôs say that you paste this text: ‚Äú[...] emphasis within a dualroute framework (e.g., Ratcliff, 1978; 2013). The idea that readers have the [...]‚Äú. The software will recognize 2 citations: Ratcliff, 1978, and Ratcliff, 2013. It will then look on the Mendeley database for the most likely papers, make some check on year, numbers/name of authors (taking into account ‚Äúet al.‚ÄĚ if present) and give back the reference list:

Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85(2) 59-108. doi:10.1037/0033-295X.85.2.59

Ratcliff, R. (2013). Parameter variability and distributional assumptions in the diffusion model. Psychological review, 120(1). doi:10.1037/a0030775

Warning and suggestions are given in case of double matching (two or more papers with same author list and year).

More information can be found in the website.

The file is completely free (even if you don’t have a MATLAB license) and everyone can change the source code (only if you have a MATLAB license, unfortunately).