CS 122D Spring 2022: Beyond SQL Data Management

Course Personnel

Instructor:

• Mike Carey
  E-Mail: ​mjcarey@ics.uci.edu
  Office hours: Ed Discussion

TAs:

• Alekhya Pyla
  E-mail: ​apyla@uci.edu
  Office hours: Ed Discussion

• Elahe Khatibi
  E-mail: ​ekhatibi@uci.edu
  Office hours: Ed Discussion

Reader:

• Sriram Rao
  E-mail: ​srirar1@uci.edu
  

Important Times & Places

Weekly Lectures:
Time: MW 3:30-4:50 PM
Place: ELH 100
Instructor: Mike

Discussion 1:
Time: W 5:00-5:50 PM
Place: DBH 1300
TA: TBD

Discussion 2:
Time: W 6:00-6:50 PM
Place: DBH 1300
TA: TBD

Discussion 3:
Time: W 7:00-7:50 PM
Place: DBH 1300
TA: TBD

Midterm Exam:
Time: Monday May 2, MW 3:30-4:50 PM
Place: Gradescope (taken in ELH 100!)

Final Exam:
Time: Monday June 6, MW 4:00-6:00 PM
Place: Gradescope (taken in ELH 100!)

Course Overview

The relational data model dates back to 1970, and relational database systems have been the dominant commercial technology for data management and data-centric application development for over three decades now. The explosion of the web, social media, and the Internet of Things are now bringing new data with challenging new requirements. This new data is richer and more diverse in structure, and in addition it is arriving in unprecedented quantities. In response, new data management technologies are appearing and being used in practice to handle today’s new data challenges.

This course surveys the new data formats and new data management and data analysis technologies that are being used for post-relational data management. Data formats covered will include those targeting semistructured data (e.g., JSON). Data management approaches surveyed will include various NoSQL technologies such as key-value stores, document database systems, and graph database systems. Surveyed data analysis technologies will include big data engines such as Hadoop, Spark, and SQL-on-Hadoop; big text search and analytics engines may also be included. Notebooks and data frames may be covered as well, time permitting, as may stream data managers and/or time-series databases. (So many systems, so little time!)

The course will be structured as a series of modules on the post-relational technologies, with chosen technologies being experienced through associated hands-on homework assignments (mini-projects). The programming language used to interact with the different systems, where needed, will be Python. Students will gain a comprehensive understanding of the basic concepts and system alternatives for post-relational data management as well as gaining hands-on experience with selected systems from the surveyed technology categories. The course is designed to be a follow-on to CS122A for those who are interested in learning about post-relational data management technologies.

Prerequisites

ICS 46, ICS 51, and CS 122A (or EECS 116) all with a grade of C or above. (B or above is strongly recommended.)

Textbooks

None! At least not the expensive kind. (Just NoSQL Distilled, an inexpensive professional book that can be read online for free though the UCI Library.) Nearly all of the materials will be accessible online for UCI students in some form, as there is no good textbook on this material today. All of the materials are listed below, and all except the ones explicitly marked optional are expected to be read. Lecture attendance is mandatory; that is, students will be held responsible for any/all information given in the lectures whether or not it appears in the course notes or elsewhere. The lectures will be made available for post-lecture reviewing in video form for students who need to hear one again or miss a lecture for some reason. The plan will be for the lecture videos and slides to become available online within about 24 hours after the class time (at least when things are going according to plan :-)).

Lecture Plan

Date	Topic	Relevant Material
M 3/28	Post-relational “escape attempts”	Ch. 1-2 NoSQL Distilled, Introducing JSON website
W 3/30	Relational model and SQL (and “beyond”!)	SQL chapters of any DB textbook, Advanced Aggregation excerpt
M 4/04	Logical DB design and E-R modeling	E-R chapter of any DB textbook
W 4/06	Scaling RDBMSs through parallelism	Parallel RDBMS paper
M 4/11	Key-value stores: Architecture & consistency	Baseball paper, Ch. 4-6, 8 NoSQL Distilled, Abadi paper
W 4/13	Column-family stores: BigTable, Cassandra	Ch. 10 NoSQL Distilled (old!)
M 4/18	Column-family stores: Cassandra (cont.)	Cassandra materials (as needed)
W 4/20	Document stores: JSON and MongoDB	Ch. 9 NoSQL Distilled (old!)
M 4/25	Document stores: MongoDB (cont. )	MongoDB materials (as needed)
W 4/27	NoSQL DB design principles	Ch. 3 NoSQL Distilled
M 5/02	Midterm Exam	3:30-4:50 PM -- in person!!!
W 5/04	Document DBs: Couchbase Server & N1QL	CB Analytics paper
M 5/09	Document DBs: Couchbase Server (cont. )	CB materials (as needed)
W 5/11	Graph DBs: Graph modeling & Neo4J	Ch. 11 NoSQL Distilled (old!)
M 5/16	Graph DBs: Neo4J (cont.)	Neo4J materials (as needed)
W 5/18	Big Data Analytics: Google, MapReduce, HDFS	Big Data Platforms paper (skim)
M 5/23	Big Data Analytics: Spark & SparkSQL	Spark Overview paper (skim)
W 5/25	Big Data Analytics: Spark & DataFrames	Databricks and Spark materials (as needed)
W 6/01	Data Stream Systems: Spark Structured Streaming	Databricks and Spark materials (as needed)
Online	Column stores, search, message streams, timeseries/IoT, ...	Optional "potpourri" video lecture (extra credit)
M 6/06	Final Exam	4:00-6:00 PM -- in person!!!

Homework Plan

HW	Available	Due Date/Time	HW Topic	Setup Info	Details	Solution
HW1	Mo 4/04	Th 4/14 (11:59 PM)	PostgreSQL	[ HW1 Setup ]	[ HW1 Details ] [ Template ]	[ HW1 Solution ]
HW2	Th 4/14	Mo 4/25 (11:59 PM)	Cassandra	[ HW2 Setup ]	[ HW2 Details ] [ Template ]	[ HW2 Solution ]
HW3	Mo 4/25	Th 5/05 (11:59 PM)	MongoDB	[ HW3 Setup ]	[ HW3 Details ] [ Template ]	[ HW3 Solution ]
HW4	Th 5/05	Mo 5/16 (11:59 PM)	CB Server	[ HW4 Setup ]	[ HW4 Details ] [ Template ]	[ HW4 Solution ]
HW5	Mo 5/16	Th 5/26 (11:59 PM)	Neo4J	[ HW5 Setup ]	[ HW5 Details ] [ Template ]	[ HW5 Solution ]
HW6	Th 5/26	Mo 6/06 (11:59 PM)	Spark	[ HW6 Setup ]	[ HW6 Details ] [ Template ]	[ HW6 Solution ]

(Note: The links above will all be broken by design until each assignment's "Available" date is reached.)

Readings and References

1. P. Sadalage and M. Fowler. NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence (1st Edition). Addison-Wesley Professional, 2012. (​https://martinfowler.com/books/nosql.html)
2. Introducing JSON, ​https://www.json.org/json-en.html.
3. A. Silbershatz, H. Korth, and S. Sudarshan, Ch. 5.5, Database System Concepts, McGraw-Hill, 2020), Advanced Aggregation in SQL​.
4. D. DeWitt and J. Gray. Parallel Database Systems: The Future of High Performance Database Systems. CACM 35, 6, June 1992. (​Parallel RDBMS)
5. A. Davoudian, L. Chen, and M. Liu. 2018. A Survey on NoSQL Stores. ACM Comp. Surv. 51, 2, April 2018. (​NoSQL Survey) (*Optional*)
6. D. Terry. Replicated Data Consistency Explained Through Baseball. CACM 58, 12, Dec. 2013. (​Baseball Paper)
7. AsterixDB 101: An ADM and SQL++ Primer, ​https://ci.apache.org/projects/asterixdb/sqlpp/primer-sqlpp.html.
8. DataStax Fundamentals, ​https://www.datastax.com/learn/cassandra-fundamentals.
9. DataStax Documentation, ​https://docs.datastax.com/.
10. MongoDB Manual, ​https://docs.mongodb.com/manual/.
11. MongoDB University, ​https://university.mongodb.com/.
12. Couchbase Docs, ​https://docs.couchbase.com/.
13. Couchbase Academy, ​https://learn.couchbase.com/.
14. D. Chamberlin, SQL++ for SQL Users: A Tutorial, Couchbase, Inc., 2018. (SQL++ ​https://asterixdb.apache.org/files/SQL_Book.pdf)
15. M.Hubail et al. Couchbase Analytics: NoETL for Scalable NoSQL Data Analysis. PVLDB 12, 12, Aug. 2019. (​CB Analytics)
16. Graph Academy, ​https://neo4j.com/graphacademy/.
17. Neo4J Documentation, ​https://neo4j.com/docs/.
18. S. Babu and H. Herodotou (2013), Massively Parallel Databases and MapReduce Systems, Foundations and Trends in Databases 5, 1, Nov. 2013. (​Big Data Platforms) (*Optional*)
19. M. Zahari et al. Apache Spark: A Unified Engine for Big Data Processing. Communications of the ACM 59, 11, Oct. 2016. (​Spark Overview)
20. Databricks Documentation, ​https://docs.databricks.com/.
21. Databricks Academy, ​https://academy.databricks.com/.
22. Databricks, Inc., The Data Engineer’s Guide to Apache Spark​, 2019.
23. Spark Structured Streaming Programming Guide, ​https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html.
24. D. Abadi, Consistency Tradeoffs in Modern Distributed Database System Design. IEEE Computer 45, 2, Feb. 2012. (​https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf)
25. Simplilearn, Introduction to Kafka, ​https://youtu.be/U4y2R3v9tlY?list=WL/. (*Optional*)
26. Elasticsearch: Getting Started, ​https://www.elastic.co/webinars/getting-started-elasticsearch/. (*Optional*)
27. Elasticsearch Reference, ​https://www.elastic.co/guide/en/elasticsearch/reference/current/. (*Optional)
28. Inside the InfluxDB Storage Engine, CMU DB Group Time Series DB Lectures, ​https://youtu.be/2SUBRE6wGiA?list=WL/. (*Optional*)

Policy Stuff

Grading Criteria and Workload

This course will include some quizzes, one midterm exam, one final exam, and a series of mini-project homework assignments appearing every 10 days or so that involve the hands-on use of the surveyed data management technologies. Because this is classified as a project course, grading will be HW-biased and will be based on a combination of the quizzes and two exams (10% + 10% + 10% = 30%), homework assignments (65%), and participation (5%).

The workload will be roughly comparable to that of CS122A, at least when this instructor teaches it. The quizzes will be graded based solely on participation, with full credit (100%) being given to each quiz earnestly attempted regardless of correctness. (Note: Participation points will be determined at the instructor's discretion and will be based on an unspecified combination of Ed Discussion analytics, in-class Q&A, and responding to various course feedback and evaluation surveys. To aid the aging instructor in the process, students who ask questions during class are encouraged to send him a short e-mail afterwards saying "Hey, I'm the student who asked you that question in class today about ...".)

Homework

Homework assignments are to be turned in by the assigned due dates/times. Details of how to turn in a given assignment will be included in each assignment's handout. The goal for the course will be for students to leave with hands-on experience with each of a variety of post-SQL data management technologies, and the homework assignments will be designed for that purpose. All of the assignments this term will be based on a common hypothetical application or web site -- so you will be working with the same data (in E-R terms), but using different technologies for different types of operations, across the whole series of mini-projects.

Due dates will be indicated on all HW assignments. Assignments will be accepted up to two days (48 hours) after the due date, but you will lose 5 points per day (out of a 100-point total) for lateness. We will not accept assignments after that time; we need to keep moving in order to make it to the end of the planned material! "Stuff happens" sometimes, so you should anticipate that reality and avoid working until the very last minute. Assignments MUST be turned in per the assignment's instructions by the indicated deadline in order to get full credit.

To ensure that you are able to do the HW assignments on time, you are strongly encouraged to have each HW assignment's setup instructions completed no later than 7 days before its due date! You will be wrestling with a number of software packages and cloud accounts, and it will simply be "your bad!" if you wait longer than that to ensure that you have the infrastructure for each HW assignment in place! (The TAs will be available to help with setup problems early on, but they will not be expected to rescue students with setup glitches that arise in the last 2-3 days before the due date for a given system's assignment.)

Collaboration

Homework assignments are to be turned in individually, but you are encouraged to pair up with a fellow student -- someone to serve as your brainstorming buddy -- for the duration of the quarter. It is okay to discuss assignments with other peers as well, e.g., to clarify the interpretation of a question or to compare thoughts on very rough approaches, but discussions of the details of your work are to stay within your team of two. You should pick this brainstorming partner at the start of the term and then stay with that same person for the remainder of the quarter. The quizzes and exams are to be taken solo, but will probably be open book, open notes, and open manual. (That's how you will work in the "real world".) Communication during exams is forbidden and will be considered an act of academic dishonesty. (Yes, unlike the "real world".)

Academic Honesty

Cheating is an area where the instructor for this course has zero patience or sympathy. You are at UCI to learn, and in this class to learn its material, not to get a grade; cheating defeats those purposes. All students will be bound by the UCI Academic Integrity and Academic Honesty policies (see ​https://aisc.uci.edu/policies/academic-integrity/index.php and ​https://www.ics.uci.edu/ugrad/policies/Academic_Honesty.php). Any student found to be involved in cheating or aiding others in cheating may be academically prosecuted to the maximum extent possible: that means you could even fail the course in its entirety. (This has happened in the past, in fact.) Just say no to cheating!!!

That being said, the instructor has been pleasantly surprised by upper division UCI students' behavior in pandemic times, at least in CS122A. As a result, he will largely be trusting students to behave with integrity and to be honest and self-policing. Again, you are presumably in this class because of the knowledge you want to gain -- we will offer it, as well as assessments of your performance -- so utilize them fully to learn! There is nothing of value to be gained by cheating, only a lost learning opportunity.

Grade Change Requests

For all of the graded assignments as well as the midterm exams, if you disagree with the grading, you may discuss your concerns with the relevant instructor (professor or TA) within two weeks after they are returned. After that, all grades will be considered final. Gradescope's regrade request feature should be used to initiate activity in all such cases -- please don't submit such requests via e-mail or Ed Discussion.

Exam Attendance

The exam dates are being provided well in advance (see above!). These dates will not be flexible and makeup exams will not be offered, so you should avoid scheduling interviews, mini-vacations, or any other activities in ways that interfere with exam-taking. All students will take the exams on the same dates and in the same time window -- on Gradescope but in person in the lecture hall -- in order to ensure that we can give, monitor, grade, and then fairly curve the entire class on each exam.

Online Discussion Forum

We will use Ed Discussion (heavily!) for online class discussions. In fact, Ed Discussion will be used in lieu of office hours for this course. Note that Ed Discussion aims to get you the help you need fast and efficiently -- from classmates, the TAs, and the instructor. Instead of emailing lecture or HW questions to the teaching staff, post them on Ed Discussion. Ed Discussion participation will contribute to your overall grade -- and misbehavior there can lead to a loss of points. You should be able to sign up for and access Ed Discussion via the CS122D home page on Canvas.

When using Ed Discussion, there are a few things to keep in mind. First, despite a national trend towards the acceptance of nastiness in social media, hurtful Ed Discussion behavior will not be tolerated -- so be kind to your classmates. Second, Ed Discussion is a great resource for asking and answering questions when used thoughtfully. You are to avoid re-asking questions that have already been asked and answered -- so you are responsible for reading others' questions and not re-asking them. With many eyes on each message, lazy question-asking can be costly, inconsiderate, and can lead to a loss of class participation points. Finally, Ed Discussion is not a venue to discuss the details of your answers to HW problems -- so it is not a place to post, request, or compare answers until after the due date plus 48 hours have passed. (Doing so risks violating the Academic Honesty Policy, and remember that you have your brainstorming buddy for such discussions.) And BTW, Ed Discussion would be a great place to post any additional info or resources that you happen to come across on the Web that you think other students may find helpful -- so feel free to post any gems that you find in your web travels for this class.

Warning: This term will mark this instructor's first use of Ed Discussion to run a class -- after many years of Piazza use -- so things may be a little rocky at first...

Lectures and Discussion Session Videos

Lecture videos (and also hopefully discussion session recordings) will be made available through Canvas and the Spring 2022 CS122D class video channel on Yuja. Last year's (Spring 2021) lectures are also still available and can be found at ​https://uci.yuja.com/V/PlayList?node=9933576&a=1583628376&autoplay=1. The lectures will follow the schedule in the Lecture Plan table above, and the Yuja channel link should take you to the latest video (with the whole collection of videos being available in the panel on the right-hand side of the screen).

Waiting List Policy

Unfortunately it is not unusual for the demand for this class to exceed the resources available to accommodate students. Wait-listed students should behave as though they are taking the class for real, coming to lectures, taking quizzes, and so on, and should sign up for and monitor Ed Discussion as well. Basically, if you are on the waiting list and want to get in, you will be required to do all of the work as though you are in the class in order to have any hope of being admitted if some students drop. Students who do not do so will simply not be admitted, as the class hits the ground running and moves too fast to allow for catch-up attempts.