Ignite Create Table

File: get_and_put.py.

• Apache Ignite Create Table
• Ignite Create Table If Not Exists
• Ignite Create Table Java
• Spark Create Table
• Apache Ignite Create Temp Table

Key-value¶

Now, that we aware of the internal structure of the Ignite SQL storage, we can create a table and put data in it using only key-value functions. For example, let us create a table to register High School students: a rough equivalent of the following SQL DDL statement: CREATE TABLE Student. From the version 2.4.0, Apache Ignite introduced a new way to connect to the Ignite cluster, which allows communication with the Ignite cluster without starting an Ignite client node. Display indexes for a table.!list. Display all active connections.!manual. Display SQLLine manual.!metadata. Invoke arbitrary metadata commands.!nickname. Create a friendly name for the connection (updates command prompt).!outputformat. Change the method for displaying SQL results.!primarykeys. Display the primary key columns for a table. The CREATE TABLE command creates a new Ignite cache and defines a SQL table on top of it. The cache stores the data in the form of key-value pairs while the table allows processing the data with SQL queries. The table will reside in the schema specified in the connection parameters. If no schema is specified, the PUBLIC schema will be used. Create table, The cache stores the data in the form of key-value pairs while the table allows processing the data with SQL queries. Apache Ignite is a memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads, delivering in-memory speeds at petabyte scale.

Type hints usage¶

File: type_hints.py

As a rule of thumb:

• when a pyignite method or function deals with a single value or key, ithas an additional parameter, like value_hint or key_hint, which acceptsa parser/constructor class,
• nearly any structure element (inside dict or list) can be replaced witha two-tuple of (said element, type hint).

Refer the Data Types section for the full listof parser/constructor classes you can use as type hints.

Scan¶

File: scans.py.

Cache’s scan() method queries allows youto get the whole contents of the cache, element by element.

Let us put some data in cache.

scan() returns a generator, that yieldstwo-tuples of key and value. You can iterate through the generated pairsin a safe manner:

Or, alternatively, you can convert the generator to dictionary in one go:

But be cautious: if the cache contains a large set of data, the dictionarymay eat too much memory!

Do cleanup¶

Destroy created cache and close connection.

SQL¶

File: sql.py.

These examples are similar to the ones given in the Apache Ignite SQLDocumentation: Getting Started.

Setup¶

First let us establish a connection.

Then create tables. Begin with Country table, than proceed with relatedtables City and CountryLanguage.

Create indexes.

Fill tables with data.

Data samples are taken from Ignite GitHub repository.

That concludes the preparation of data. Now let us answer some questions.

What are the 10 largest cities in our data sample (population-wise)?¶

The sql() method returns a generator,that yields the resulting rows.

What are the 10 most populated cities throughout the 3 chosen countries?¶

If you set the include_field_names argument to True, thesql() method will generate a list ofcolumn names as a first yield. You can access field names with Python built-innext function.

Display all the information about a given city¶

Finally, delete the tables used in this example with the following queries:

Complex objects¶

File: binary_basics.py.

Complex object (that is often called ‘Binary object’) is an Ignite datatype, that is designed to represent a Java class. It have the followingfeatures:

• have a unique ID (type id), which is derives from a class name (type name),
• have one or more associated schemas, that describes its inner structure (theorder, names and types of its fields). Each schema have its own ID,
• have an optional version number, that is aimed towards the end usersto help them distinguish between objects of the same type, serializedwith different schemas.

Unfortunately, these distinctive features of the Complex object have few to nomeaning outside of Java language. Python class can not be defined by its name(it is not unique), ID (object ID in Python is volatile; in CPython it is justa pointer in the interpreter’s memory heap), or complex of its fields (theydo not have an associated data types, moreover, they can be added or deletedin run-time). For the pyignite user it means that for all purposesof storing native Python data it is better to use IgniteCollectionObjector MapObject data types.

However, for interoperability purposes, pyignite has a mechanism of creatingspecial Python classes to read or write Complex objects. These classes havean interface, that simulates all the features of the Complex object: type name,type ID, schema, schema ID, and version number.

Assuming that one concrete class for representing one Complex object canseverely limit the user’s data manipulation capabilities, all thefunctionality said above is implemented through the metaclass:GenericObjectMeta. This metaclass is usedautomatically when reading Complex objects.

Here you can see how GenericObjectMeta usesattrs package internally for creating nice __init__() and __repr__()methods.

You can reuse the autogenerated class for subsequent writes:

GenericObjectMeta can also be used directlyfor creating custom classes:

Note how the Person class is defined. schema is aGenericObjectMeta metaclass parameter.Another important GenericObjectMeta parameteris a type_name, but it is optional and defaults to the class name (‘Person’in our example).

Note also, that Person do not have to define its own attributes, methods andproperties (pass), although it is completely possible.

Now, when your custom Person class is created, you are ready to send datato Ignite server using its objects. The client will implicitly register yourclass as soon as the first Complex object is sent. If you intend to use yourcustom class for reading existing Complex objects’ values before all, you mustregister said class explicitly with your client:

Now, when we dealt with the basics of pyignite implementation of ComplexObjects, let us move on to more elaborate examples.

Read¶

File: read_binary.py.

Ignite SQL uses Complex objects internally to represent keys and rowsin SQL tables. Normally SQL data is accessed via queries (see SQL),so we will consider the following example solely for the demonstrationof how Binary objects (not Ignite SQL) work.

In the previous examples we have created some SQL tables.Let us do it again and examine the Ignite storage afterwards.

We can see that Ignite created a cache for each of our tables. The caches areconveniently named using ‘SQL_<schema name>_<table name>’ pattern.

Now let us examine a configuration of a cache that contains SQL datausing a settings property.

The values of value_type_name and key_type_name are names of the binarytypes. The City table’s key fields are stored using key_type_name type,and the other fields − value_type_name type.

Now when we have the cache, in which the SQL data resides, and the namesof the key and value data types, we can read the data without using SQLfunctions and verify the correctness of the result.

What we see is a tuple of key and value, extracted from the cache. Both keyand value are represent Complex objects. The dataclass names are the sameas the value_type_name and key_type_name cache settings. The objects’fields correspond to the SQL query.

Create¶

File: create_binary.py.

Now, that we aware of the internal structure of the Ignite SQL storage,we can create a table and put data in it using only key-value functions.

For example, let us create a table to register High School students:a rough equivalent of the following SQL DDL statement:

These are the necessary steps to perform the task.

1. Create table cache.

2. Define Complex object data class.

3. Insert row.

Now let us make sure that our cache really can be used with SQL functions.

Note, however, that the cache we create can not be dropped with DDL command.

It should be deleted as any other key-value cache.

Migrate¶

File: migrate_binary.py.

Suppose we have an accounting app that stores its data in key-value format.Our task would be to introduce the following changes to the original expensevoucher’s format and data:

• rename date to expense_date,
• add report_date,
• set report_date to the current date if reported is True, None if False,
• delete reported.

First get the vouchers’ cache.

If you do not store the schema of the Complex object in code, you can obtainit as a dataclass property withquery_binary_type() method.

Let us modify the schema and create a new Complex object class with an updatedschema.

Now migrate the data from the old schema to the new one.

At this moment all the fields, defined in both of our schemas, can beavailable in the resulting binary object, depending on which schema was usedwhen writing it using put() or similar methods.Ignite Binary API do not have the method to delete Complex object schema;all the schemas ever defined will stay in cluster until its shutdown.

This versioning mechanism is quite simple and robust, but it have itslimitations. The main thing is: you can not change the type of the existingfield. If you try, you will be greeted with the following message:

`org.apache.ignite.binary.BinaryObjectException:Wrongvaluehasbeenset[typeName=SomeType,fieldName=f1,fieldType=String,assignedValueType=int]`

As an alternative, you can rename the field or create a new Complex object.

Failover¶

File: failover.py.

When connection to the server is broken or timed out,Client object propagates an original exception(OSError or SocketError), but keeps its constructor’s parameters intactand tries to reconnect transparently.

When there’s no way for Client to reconnect, itraises a special ReconnectError exception.

The following example features a simple node list traversal failover mechanism.Gather 3 Ignite nodes on localhost into one cluster and run:

Then try shutting down and restarting nodes, and see what happens.

Client reconnection do not require an explicit user action, like callinga special method or resetting a parameter. Note, however, that reconnectionis lazy: it happens only if (and when) it is needed. In this example,the automatic reconnection happens, when the script checks upon the lastsaved value:

It means that instead of checking the connection status it is better forpyignite user to just try the supposed data operations and catchthe resulting exception.

connect() method accepts anyiterable, not just list. It means that you can implement any reconnectionpolicy (round-robin, nodes prioritization, pause on reconnect or gracefulbackoff) with a generator.

pyignite comes with a sampleRoundRobin generator. In the aboveexample try to replace

with

The client will try to reconnect to node 1 after node 3 is crashed, then tonode 2, et c. At least one node should be active for theRoundRobin to work properly.

SSL/TLS¶

There are some special requirements for testing SSL connectivity.

The Ignite server must be configured for securing the binary protocol port.The server configuration process can be split up into these basic steps:

1. Create a key store and a trust store using Java keytool. When creatingthe trust store, you will probably need a client X.509 certificate. Youwill also need to export the server X.509 certificate to include in theclient chain of trust.
2. Turn on the SslContextFactory for your Ignite cluster according to thisdocument: Securing Connection Between Nodes.
3. Tell Ignite to encrypt data on its thin client port, using the settings forClientConnectorConfiguration. If you only want to encrypt connection,not to validate client’s certificate, set sslClientAuth property tofalse. You’ll still have to set up the trust store on step 1 though.

Client SSL settings is summarized here:Client.

To use the SSL encryption without certificate validation just use_ssl.

To identify the client, create an SSL keypair and a certificate withopenssl command and use them in this manner:

Apache Ignite Create Table

To check the authenticity of the server, get the server certificate orcertificate chain and provide its path in the ssl_ca_certfile parameter.

You can also provide such parameters as the set of ciphers (ssl_ciphers) andthe SSL version (ssl_version), if the defaults(ssl._DEFAULT_CIPHERS and TLS 1.1) do not suit you.

Password authentication¶

To authenticate you must set authenticationEnabled property to true andenable persistance in Ignite XML configuration file, as described inAuthentication section of Ignite documentation.

Be advised that sending credentials over the open channel is greatlydiscouraged, since they can be easily intercepted. Supplying credentialsautomatically turns SSL on from the client side. It is highly recommendedto secure the connection to the Ignite server, as describedin SSL/TLS example, in order to use password authentication.

Then just supply username and password parameters toClient constructor.

If you still do not wish to secure the connection is spite of the warning,then disable SSL explicitly on creating the client object:

Note, that it is not possible for Ignite thin client to obtain the cluster’sauthentication settings through the binary protocol. Unexpected credentialsare simply ignored by the server. In the opposite case, the user is greetedwith the following message:

What if you only had 5 minutes and 20 slides to wow a roomful of people? That’s the premise of an Ignite talk. Your slides auto-advance every 15 seconds. That means that you have to choose your words carefully, create slides with a lot of visual impact and be very entertaining. Think of it as an abbreviated TED talk.

According to Igniteshow.com, “Ignite is a fast-paced geek event started by Brady Forrest, Technology Evangelist for O’Reilly Media, and Bre Pettis of Makerbot.com, formerly of MAKE Magazine. Speakers are given 20 slides, each shown for 15 seconds, giving each speaker 5 minutes of fame. The first Ignite took place in Seattle in 2006, and since then the event has become an international phenomenon, with gatherings all over the world.”

Ignite Create Table If Not Exists

You don’t have to be at an Ignite event to give an Ignite talk. Anytime you want to communicate quickly and in an entertaining way is a great time for an Ignite presentation. Here’s how to develop one.

1. Start with an outline

Every presentation you create should start with an outline. Too many times people start their presentations by creating slides, which is a little like shooting a movie with no script: you might get some good visuals but your story is all over the place. Your outline should be done either on paper or with text-editing software. Anything but PowerPoint! You want to concentrate on story structure right now. The slides come later.

Your outline is the foundation of your speech. It should have a strong opening premise, much like the first paragraph in a newspaper or magazine article. You want to grab people’s attention and to tell them what you’re going to tell them. The body of the outline is your supporting information. Stories, facts, statistics…everything that lends credence to your introduction. Last comes the big finish: call to action, summation of what you’ve talked about, thoughtful question for the audience, etc.

2. Write your script

After you’ve polished your outline, it’s time to write your script. Write exactly the same way as you usually talk, but leave out unexplained jargon, highly technical terms that most people wouldn’t understand, gigantic words you seldom use, etc.. Make your tone conversational, as if you’re just speaking with a friend.

When you’re happy with your script, then create a blank 20-slide PowerPoint presentation. Cut and paste your script onto the slides. Make any edits you think you need, but don’t do any layout or add any images. We’re still just dealing with words at this point.

3. Fine tune the timing

Go to Slide Show/Rehearse Timings and start reading!

Ignite Create Table Java

Tap the space bar on your keyboard to advance after you read each slide. At the end of your presentation, you will see a prompt telling you how long your narration was and asking if you want to save the new slide timings. Click “Yes.”

Spark Create Table

On your first runthrough, you’ll notice that some slides go on for too long, others not long enough. Edit your speech by adding, deleting and rearranging text until reading each slide takes 13–14 seconds. Once you’ve gotten your timing right, print out your presentation. This will be your rehearsal and recording script.

4. Design your slides

Ignite presentations tend to contain large pictures, simple charts and graphs, and a minimal amount of text without any bullet points. Sometimes, an Ignite presentation will have only one photo per slide. Here are some examples:

Using your script as a guide, add visuals to the slides. Some slides could be all text, others could be all images. How your slides look will depend entirely on your script. An Ignite deck is not a teleprompter, so avoid the temptation to put too much information on your slides.

Don’t use animation that is activated by mouse clicks. All animation should be set to run automatically. Any animation you include should be uncomplicated because the real star of an Ignite presentation is the speaker…you!

5. Set your slide times

Click on the Transitions tab on the Ribbon. Select Fade (#1, above). In the Timing group, deselect “On Mouse Click” and select “After.” Type 15 in the box (#2, above). Then click “Apply to All (#3, above). Now you’re ready to…

Apache Ignite Create Temp Table

6. Practice!

Practice, practice, practice, practice, practice, practice, practice, practice, practice, practice, practice, practice, practice! Am I making myself clear?

Your delivery should be smooth, unhurried, and seem unrehearsed. This comes only with a complete familiarity with what you’re going to say.

Now, go get ’em, killer! This is one presentation your audience will remember for a long time!