Added a web script to dump a full dataset for a publication

README.md

@@ -49,6 +49,7 @@ Version 1.0, 8.3.17 (ak):
 ```
 readme.md       This file
 config.py       Site dependant configuration file (in GIT as config.py.sample)
+scopus-rawdata.py 		Display the complete dataset for a publication in Scopus
 my_scopus.py	List of scopus author ids
 ak_scopus.py	Functions to access scopus
 ak_wordpress.py Functions to creates Wordpress posts + comments
@@ -66,6 +67,8 @@ info            Documentation, website, etc (not in GIT)
 log		Log file use scopus-publications-<hostname>.log 
 ```
 
+
+
 ## Usage
 
 1. Go to Scopus and retrieve the scopus author ids for the scientists in your group.
@@ -85,6 +88,7 @@ python -W ignore scopus-get-publications.py
 Note: The -W ignore flag might be necessary if the INSERT IGNORE causes warnings.
 
 Example run:
+
 ```
 ufo:~/scopus # python -W ignore scopus-get-publications.py 
 
@@ -119,6 +123,22 @@ NNewCites  = 0
 Runtime    = 0:00:11.496362
 ```
 
+
+### Dump datasets
+
+Use the Python script ```scopus-rawdata.py``` to get the full dataset of a certain publication. The script is intended for usage in a web browser. Most browsers have pulgins to print formated JSON output. 
+
+Example: 
+
+```
+http://localhost/~kopmann/scopus/scopus-rawdata.py?85042753214
+```
+
+
+![Scrennshot](scopus-rawdata.png)
+
+
+
 ## Further enhancements
 
 Todo:
@@ -134,6 +154,7 @@ like UFO or may be also later for the DTS program.
 bibtex definition and upload on a server!?
 This would have the nice effect, that all student work is organized systematically!!!
 
+- keywords
 
 
 ## Structure of the database
@@ -346,6 +367,295 @@ Sample data from Scopus:
 }
 ```
 
+## Keywords in Scopus
+
+
+There are two types of keywords known in Scopus. Not all datasets have both fields. Most of them have the index terms (field: idxterms) that seem to be selected by Scopus. Many have additional keywords given by the author (field authkeywords. 
+
+Often author keywords are used as index term. Index term always start with a capital letter, while author keywords are less systematic. Somtimes author keywords are very long, with brackets and semicolon.
+
+The question is how to use this keywords. Are there some that apprear in more than one article. This one might be used to find similar articles in Wordpress later!?
+
+Should I take both keywords, or should I use the index terms, although I don't know where they come from? Does it make sense to modify the keywords once there are in Wordpress???
+
+Für wordpress macht es wahrscheinlich Sinn die Keywords auf zwei bis drei Wörter zu beschränken und die anderen einfach weg zu lassen, da sie doch nicht zweimal auftreten werden!
+
+
+```
+ipekopmann2:scopus kopmann$ python test-scopus.py 
+==> Index terms:
+Drying at ambient conditions
+High complexity
+Line of Sight
+On-line analysis
+Parallel computing architecture
+Particle image velocimetries
+Polymer Coating
+Thin polymer films
+!!! Error encoding title of publication !!!
+Cavadini P., Weinhold H., Tonsmann M., Chilingaryan S., Kopmann A., Lewkowicz A., Miao C., Scharfer P., Schabel W.: Investigation of the flow structure in thin polymer films using 3D µPTV enhanced by GPU, Experiments in Fluids, 59, 4 (2018-04-01) 61, doi:10.1007/s00348-017-2482-z (cited 0 times).
+
+Chilingarian A., Chilingaryan S., Karapetyan T., Kozliner L., Khanikyants Y., Hovsepyan G., Pokhsraryan D., Soghomonyan S.: On the initiation of lightning in thunderclouds, Scientific Reports, 7, 1 (2017-12-01) 1371, doi:10.1038/s41598-017-01288-0 (cited 3 times).
+
+==> Index terms:
+Free space propagation
+High speed imaging
+Parallelizations
+simulation
+Xray imaging
+==> Keywords:
+coherence
+free-space propagation
+high-speed imaging
+parallelization
+simulation
+synchrotron radiation
+X-ray imaging
+Farago T., Mikulik P., Ershov A., Vogelgesang M., Hanschke D., Baumbach T.: Syris: A flexible and efficient framework for X-ray imaging experiments simulation, Journal of Synchrotron Radiation, 24, 6 (2017-11-01) 1283-1295, doi:10.1107/S1600577517012255 (cited 0 times).
+
+==> Index terms:
+Direct communications
+High quality reconstruction
+High speed computing
+High-performance computing resources
+Online data processing
+Reconstruction algorithms
+Regularization methods
+Spatial and temporal resolutions
+Kopmann A., Chilingaryan S., Vogelgesang M., Dritschler T., Shkarin A., Shkarin R., Dos Santos Rolo T., Farago T., Van De Kamp T., Balzer M., Caselle M., Weber M., Baumbach T.: UFO - A scalable platform for high-speed synchrotron X-ray imaging, 2016 IEEE Nuclear Science Symposium, Medical Imaging Conference and Room-Temperature Semiconductor Detector Workshop, NSS/MIC/RTSD 2016, 2017-January (2017-10-16) 8069895, doi:10.1109/NSSMIC.2016.8069895 (cited 0 times).
+
+==> Index terms:
+Computing (architecture, farms, GRID for recording, storage, archiving, and distribution of data)
+Data processing methods
+Dedicated hardware
+Floating point operations
+Hardware and software
+Hough Transformation
+Large Hadron collider LHC
+Modified algorithms
+==> Keywords:
+Computing (architecture, farms, GRID for recording, storage, archiving, and distribution of data)
+Trigger algorithms; Data processing methods
+Trigger concepts and systems (hardware and software)
+Mohr H., Dritschler T., Ardila L.E., Balzer M., Caselle M., Chilingaryan S., Kopmann A., Rota L., Schuh T., Vogelgesang M., Weber M.: Evaluation of GPUs as a level-1 track trigger for the High-Luminosity LHC, Journal of Instrumentation, 12, 4 (2017-04-21) C04019, doi:10.1088/1748-0221/12/04/C04019 (cited 0 times).
+
+==> Index terms:
+Advanced mezzanine cards
+Board support packages
+Commercial components
+Component based approach
+Detector control systems
+Front end electronics
+Parallel-computing environment
+Research and development
+==> Keywords:
+Data acquisition concepts
+Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases)
+Image reconstruction in medical imaging
+Software architectures (event data models, frameworks and databases)
+Kaever P., Balzer M., Kopmann A., Zimmer M., Rongen H.: The Common Data Acquisition Platform in the Helmholtz Association, Journal of Instrumentation, 12, 4 (2017-04-03) C04004, doi:10.1088/1748-0221/12/04/C04004 (cited 0 times).
+
+==> Index terms:
+Computing nodes
+Direct communications
+Event building
+Future generations
+Hardware and software
+High-level triggers
+Internal memory
+Trigger systems
+==> Keywords:
+Data acquisition concepts
+Trigger concepts and systems (hardware and software)
+Caselle M., Perez L.E.A., Balzer M., Dritschler T., Kopmann A., Mohr H., Rota L., Vogelgesang M., Weber M.: A high-speed DAQ framework for future high-level trigger and event building clusters, Journal of Instrumentation, 12, 3 (2017-03-06) C03015, doi:10.1088/1748-0221/12/03/C03015 (cited 1 times).
+
+==> Index terms:
+Accelerator physics
+Coherent synchrotron radiation
+Data acquisition system
+Electronic detectors
+High repetition rate
+Radiation accelerators
+Sampling accuracies
+Terahertz detectors
+==> Keywords:
+Data acquisition concepts
+Electronic detector readout concepts (solid-state)
+Instrumentation for synchrotron radiation accelerators
+Caselle M., Perez L.E.A., Balzer M., Kopmann A., Rota L., Weber M., Brosi M., Steinmann J., Brundermann E., Muller A.-S.: KAPTURE-2. A picosecond sampling system for individual THz pulses with high repetition rate, Journal of Instrumentation, 12, 1 (2017-01-16) C01040, doi:10.1088/1748-0221/12/01/C01040 (cited 0 times).
+
+==> Index terms:
+Conventional approach
+Micro-tomography
+Phase contrasts
+Photosynthetic activity
+Physiological activity
+Picea Abies (L.) Karst
+Spruce
+Tracheid
+==> Keywords:
+Absorption contrast
+Microtomography
+Phase contrast
+Spruce
+Synchrotron radiation
+Tracheid
+Wood
+!!! Error encoding title of publication !!!
+Lautner S., Lenz C., Hammel J., Moosmann J., Kuhn M., Caselle M., Vogelgesang M., Kopmann A., Beckmann F.: Using SRμCT to define water transport capacity in Picea abies, Proceedings of SPIE - The International Society for Optical Engineering, 10391 (2017-01-01) 1039118, doi:10.1117/12.2287221 (cited 3 times).
+
+==> Index terms:
+Data catalog
+Insect head
+Interactive interfaces
+Scientific data
+Semi-automatic segmentation
+Synchrotron x rays
+Visual data
+Web visualization
+==> Keywords:
+3D web visualization
+Cooperative data analysis
+Data catalog
+Insect head
+Interactive interfaces
+Semi-automatic segmentation
+Synchrotron X-ray micro computed tomography
+Virtual Reality for scientific data
+Visual data browsing
+Web portal for scientific data
+!!! Error encoding title of publication !!!
+Schmelzle S., Heethoff M., Heuveline V., Losel P., Becker J., Beckmann F., Schluenzen F., Hammel J.U., Kopmann A., Mexner W., Vogelgesang M., Jerome N.T., Betz O., Beutel R., Wipfler B., Blanke A., Harzsch S., Hornig M., Baumbach T., Van De Kamp T.: The NOVA project: Maximizing beam time efficiency through synergistic analyses of SRμCT data, Proceedings of SPIE - The International Society for Optical Engineering, 10391 (2017-01-01) 103910P, doi:10.1117/12.2275959 (cited 0 times).
+
+==> Index terms:
+Astroparticle physics
+DAQ system
+Dark matter
+Data acquisition system
+Direct search
+Germaniums (Ge)
+High-rate channels
+Underground laboratory
+Bergmann T., Balzer M., Bormann D., Chilingaryan S.A., Eitel K., Kleifges M., Kopmann A., Kozlov V., Menshikov A., Siebenborn B., Tcherniakhovski D., Vogelgesang M., Weber M.: A scalable DAQ system with high-rate channels and FPGA- and GPU-Trigger for the dark matter experiment EDELWEISS-III, 2015 IEEE Nuclear Science Symposium and Medical Imaging Conference, NSS/MIC 2015 (2016-10-03) 7581841, doi:10.1109/NSSMIC.2015.7581841 (cited 2 times).
+
+==> Index terms:
+Computational power
+Digital electronic circuits
+Direct memory access
+Direct memory transfers
+Hardware and software
+High performance computing
+High-performance computing applications
+Intrinsic parallelisms
+==> Keywords:
+Data acquisition concepts
+Digital electronic circuits
+Trigger concepts and systems (hardware and software)
+Rota L., Vogelgesang M., Perez L.E.A., Caselle M., Chilingaryan S., Dritschler T., Zilio N., Kopmann A., Balzer M., Weber M.: A high-throughput readout architecture based on PCI-Express Gen3 and DirectGMA technology, Journal of Instrumentation, 11, 2 (2016-02-12) P02007, doi:10.1088/1748-0221/11/02/P02007 (cited 5 times).
+
+==> Index terms:
+Data acquisition system
+Experiment platforms
+GPU computing
+High-throughput data
+OpenCL
+Programmable hardware
+Real-time operation
+Synchrotron beamlines
+==> Keywords:
+Data acquisition
+data processing
+FPGA
+FPGA-GPU communication
+GPU computing
+OpenCL
+Vogelgesang M., Rota L., Perez L.E.A., Caselle M., Chilingaryan S., Kopmann A.: High-throughput data acquisition and processing for real-time X-ray imaging, Proceedings of SPIE - The International Society for Optical Engineering, 9967 (2016-01-01) 996715, doi:10.1117/12.2237611 (cited 0 times).
+
+Ametova E., Ferrucci M., Chilingaryan S., McCarthy M., Dewulf W.: Uncertainty quantification in dimensional measurements by computed tomography due to uncertainty in data acquisition geometrical parameters, Proceedings - ASPE 2016 Annual Meeting (2016-01-01) 287-292 (cited 0 times).
+
+==> Index terms:
+Architecture-based
+Building blockes
+Efficient construction
+High-level control systems
+Laminography
+Real time images
+Realtime processing
+Synchrotron radiation facility
+High-throughput data
+Work-flow systems
+==> Keywords:
+control
+laminography
+tomography
+Vogelgesang M., Farago T., Morgeneyer T.F., Helfen L., Dos Santos Rolo T., Myagotin A., Baumbach T.: Real-time image-content-based beamline control for smart 4D X-ray imaging, Journal of Synchrotron Radiation, 23 (2016-01-01) 1254-1263, doi:10.1107/S1600577516010195 (cited 6 times).
+
+==> Index terms:
+Armenia
+Energy spectra
+Energy thresholds
+Enhancements
+Ground
+Solar cosmic rays
+==> Keywords:
+Atmospheric electricity
+Enhancements
+Ground
+Particle detectors
+Thunderstorm
+Chilingarian A., Chilingaryan S., Hovsepyan G.: Calibration of particle detectors for secondary cosmic rays using gamma-ray beams from thunderclouds, Astroparticle Physics, 69 (2015-09-01) 37-43, doi:10.1016/j.astropartphys.2015.03.011 (cited 3 times).
+
+==> Keywords:
+atmospheric electricity
+radiation in atmosphere
+Chilingarian A., Chilingaryan S., Reymers A.: Atmospheric discharges and particle fluxes, Journal of Geophysical Research A: Space Physics, 120, 7 (2015-07-01) 5845-5853, doi:10.1002/2015JA021259 (cited 2 times).
+
+==> Index terms:
+Data throughput
+Direct memory access
+High-speed data
+PCI Express
+Readout Electronics
+==> Keywords:
+Data Acquisition
+direct memory access
+FPGA
+high data throughput
+high speed data streaming applications
+PCI express
+readout electronics
+Rota L., Caselle M., Chilingaryan S., Kopmann A., Weber M.: A PCIe DMA Architecture for Multi-Gigabyte per Second Data Transmission, IEEE Transactions on Nuclear Science, 62, 3 (2015-06-01) 972-976, doi:10.1109/TNS.2015.2426877 (cited 9 times).
+
+==> Index terms:
+CMOS image sensor
+Complementary metal-oxide-semiconductor sensor (CMOS)
+Data acquisition system
+Embedded processing
+Material science
+Smart cameras
+Synchrotron radiation facility
+Temporal evolution
+==> Keywords:
+CMOS image sensors
+control systems
+data processing
+FPGAs
+smart cameras
+Stevanovic U., Caselle M., Cecilia A., Chilingaryan S., Farago T., Gasilov S., Herth A., Kopmann A., Vogelgesang M., Balzer M., Baumbach T., Weber M.: A control system and streaming DAQ platform with image-based trigger for X-ray imaging, IEEE Transactions on Nuclear Science, 62, 3 (2015-06-01) 911-918, doi:10.1109/TNS.2015.2425911 (cited 2 times).
+
+==> Index terms:
+Coherent synchrotron radiation
+Continuous sampling
+Data throughput
+Intrinsic response
+Peak amplitude
+Synchrotron light source
+Terahertz pulse
+Thin film detectors
+Caselle M., Brosi M., Chilingaryan S., Dritschler T., Judin V., Kopmann A., Mueller A.-S., Raasch J., Smale N.J., Steinmann J., Vogelgesang M., Wuensch S., Siegel M., Weber M.: An ultra-fast digitizer with picosecond sampling time for Coherent Synchrotron Radiation, 2014 19th IEEE-NPSS Real Time Conference, RT 2014 - Conference Records (2015-04-28) 7097535, doi:10.1109/RTC.2014.7097535 (cited 2 times).
+```
+
+
 
 ## Installation of tools
 

scopus-rawdata.py

@@ -0,0 +1,36 @@
+#!/usr/bin/python
+#
+# Access Scopus database
+#
+
+import sys
+import requests
+import json
+
+#from config import MY_API_KEY
+MY_API_KEY = "14d431d052c2caf5e9c4b1ab7de7463d"
+
+# Examples of our publications
+#SCOPUS_ID = "SCOPUS_ID:85032685965"
+#SCOPUS_ID = "SCOPUS_ID:84940537475"
+
+
+if (len(sys.argv) > 1):
+   SCOPUS_ID = "SCOPUS_ID:" + sys.argv[1]
+
+   url = ("http://api.elsevier.com/content/abstract/scopus_id/"
+           + SCOPUS_ID)
+   resp = requests.get(url,
+                headers={'Accept':'application/json',
+                        'X-ELS-APIKey': MY_API_KEY})
+   results = json.loads(resp.text.encode('utf-8'))
+
+   print "Content-type: text/html\n\n";
+   print(json.dumps(results))
+
+else: 
+   print "Content-type: text/html\n\n";
+   print "Usage: " + sys.argv[0] + "?12334  (put SCOPUS_ID here)"
+
+ 
+