• Log in

Send a custom span event

5 min

lab

This procedure is part of a lab that teaches you how to instrument your application with OpenTelemetry.

Each procedure in the lab builds upon the last, so make sure you've completed the last procedure, View your OpenTelemetry data in New Relic, before starting this one.

You've decided you don't want the Python OpenTelemetry SDK to automatically record exceptions as exception span events, because they're not really errors in the database application. These are expected exceptions based on user behavior. Here, you modify your code to record a custom span event, rather than automatically collecting an exception span event.

Modify your instrumentation

Step 1 of 5

In the terminal window that's running your simulator, press <CTRL-C>.

You should see your simulator shut down. Now you can update your app logic to add custom span events.

Step 2 of 5

In db.py, modify your span context managers to not record exceptions as span events:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8
9
provider = TracerProvider(
10
resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
BatchSpanProcessor(
14
OTLPSpanExporter(compression=Compression.Gzip)
15
)
16
)
17
trace.set_tracer_provider(provider)
18
19
tracer = trace.get_tracer(__name__)
20
21
class DuplicateKeyError(Exception):
22
pass
23
24
class KeyDoesNotExistError(Exception):
25
pass
26
27
db = {}
28
29
def read(key):
30
"""Read key from the database."""
31
global db
32
33
with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
try:
35
value = db[key]
36
logging.debug("Successful read")
37
span.set_attribute("key", key)
38
return value
39
except KeyError as ke:
40
msg = f"Key `{key}` doesn't exist"
41
logging.debug(msg)
42
raise KeyDoesNotExistError(msg)
43
44
def create(key, value):
45
"""Write key:value to the database."""
46
global db
47
48
with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
49
if key in db:
50
msg = f"Key `{key}` already exists"
51
logging.debug(msg)
52
raise DuplicateKeyError(msg)
53
54
db[key] = value
55
logging.debug("Successful create")
56
span.set_attribute("key", key)
57
return value
58
59
def update(key, value):
60
"""Update key in the database."""
61
global db
62
63
with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
64
if key in db:
65
db[key] = value
66
logging.debug("Successful update")
67
span.set_attribute("key", key)
68
return value
69
70
msg = f"Key `{key}` doesn't exist"
71
logging.debug(msg)
72
raise KeyDoesNotExistError(msg)
73
74
def delete(key):
75
"""Delete key from the database."""
76
global db
77
78
with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
79
if key in db:
80
del db[key]
81
logging.debug("Successful delete")
82
span.set_attribute("key", key)
83
return True
84
85
return False
db.py

Your code will no longer save an exception span event on your spans. However, you still want to know how many times your users attempt to perform these actions against your database. To do this, record your own custom span events.

Tip

Notice that you didn't update the context manager for delete(). This is because the logic in this function doesn't use an exception to indicate the state of the database. Any exception that is raised in the underlying code here, will be a real error. It still makes sense to let the SDK manage real errors.

Step 3 of 5

Record a span event for read():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8
9
provider = TracerProvider(
10
resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
BatchSpanProcessor(
14
OTLPSpanExporter(compression=Compression.Gzip)
15
)
16
)
17
trace.set_tracer_provider(provider)
18
19
tracer = trace.get_tracer(__name__)
20
21
class DuplicateKeyError(Exception):
22
pass
23
24
class KeyDoesNotExistError(Exception):
25
pass
26
27
db = {}
28
29
def read(key):
30
"""Read key from the database."""
31
global db
32
33
with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
try:
35
value = db[key]
36
logging.debug("Successful read")
37
span.set_attribute("key", key)
38
return value
39
except KeyError as ke:
40
msg = f"Key `{key}` doesn't exist"
41
logging.debug(msg)
42
span.add_event("read_key_dne", {"key": key})
43
raise KeyDoesNotExistError(msg)
44
45
def create(key, value):
46
"""Write key:value to the database."""
47
global db
48
49
with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
if key in db:
51
msg = f"Key `{key}` already exists"
52
logging.debug(msg)
53
raise DuplicateKeyError(msg)
54
55
db[key] = value
56
logging.debug("Successful create")
57
span.set_attribute("key", key)
58
return value
59
60
def update(key, value):
61
"""Update key in the database."""
62
global db
63
64
with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
65
if key in db:
66
db[key] = value
67
logging.debug("Successful update")
68
span.set_attribute("key", key)
69
return value
70
71
msg = f"Key `{key}` doesn't exist"
72
logging.debug(msg)
73
raise KeyDoesNotExistError(msg)
74
75
def delete(key):
76
"""Delete key from the database."""
77
global db
78
79
with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
80
if key in db:
81
del db[key]
82
logging.debug("Successful delete")
83
span.set_attribute("key", key)
84
return True
85
86
return False
db.py

If you try to read a key that does not exist, your application adds a span event on the span.

Step 4 of 5

Record a span event for create():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8
9
provider = TracerProvider(
10
resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
BatchSpanProcessor(
14
OTLPSpanExporter(compression=Compression.Gzip)
15
)
16
)
17
trace.set_tracer_provider(provider)
18
19
tracer = trace.get_tracer(__name__)
20
21
class DuplicateKeyError(Exception):
22
pass
23
24
class KeyDoesNotExistError(Exception):
25
pass
26
27
db = {}
28
29
def read(key):
30
"""Read key from the database."""
31
global db
32
33
with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
try:
35
value = db[key]
36
logging.debug("Successful read")
37
span.set_attribute("key", key)
38
return value
39
except KeyError as ke:
40
msg = f"Key `{key}` doesn't exist"
41
logging.debug(msg)
42
span.add_event("read_key_dne", {"key": key})
43
raise KeyDoesNotExistError(msg)
44
45
def create(key, value):
46
"""Write key:value to the database."""
47
global db
48
49
with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
if key in db:
51
msg = f"Key `{key}` already exists"
52
logging.debug(msg)
53
span.add_event("create_key_exists", {"key": key})
54
raise DuplicateKeyError(msg)
55
56
db[key] = value
57
logging.debug("Successful create")
58
span.set_attribute("key", key)
59
return value
60
61
def update(key, value):
62
"""Update key in the database."""
63
global db
64
65
with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
66
if key in db:
67
db[key] = value
68
logging.debug("Successful update")
69
span.set_attribute("key", key)
70
return value
71
72
msg = f"Key `{key}` doesn't exist"
73
logging.debug(msg)
74
raise KeyDoesNotExistError(msg)
75
76
def delete(key):
77
"""Delete key from the database."""
78
global db
79
80
with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
81
if key in db:
82
del db[key]
83
logging.debug("Successful delete")
84
span.set_attribute("key", key)
85
return True
86
87
return False
db.py

If you try to create a key that already exists, your application adds a span event on the span.

Step 5 of 5

Record a span event for update():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8
9
provider = TracerProvider(
10
resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
BatchSpanProcessor(
14
OTLPSpanExporter(compression=Compression.Gzip)
15
)
16
)
17
trace.set_tracer_provider(provider)
18
19
tracer = trace.get_tracer(__name__)
20
21
class DuplicateKeyError(Exception):
22
pass
23
24
class KeyDoesNotExistError(Exception):
25
pass
26
27
db = {}
28
29
def read(key):
30
"""Read key from the database."""
31
global db
32
33
with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
try:
35
value = db[key]
36
logging.debug("Successful read")
37
span.set_attribute("key", key)
38
return value
39
except KeyError as ke:
40
msg = f"Key `{key}` doesn't exist"
41
logging.debug(msg)
42
span.add_event("read_key_dne", {"key": key})
43
raise KeyDoesNotExistError(msg)
44
45
def create(key, value):
46
"""Write key:value to the database."""
47
global db
48
49
with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
if key in db:
51
msg = f"Key `{key}` already exists"
52
logging.debug(msg)
53
span.add_event("create_key_exists", {"key": key})
54
raise DuplicateKeyError(msg)
55
56
db[key] = value
57
logging.debug("Successful create")
58
span.set_attribute("key", key)
59
return value
60
61
def update(key, value):
62
"""Update key in the database."""
63
global db
64
65
with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
66
if key in db:
67
db[key] = value
68
logging.debug("Successful update")
69
span.set_attribute("key", key)
70
return value
71
72
msg = f"Key `{key}` doesn't exist"
73
logging.debug(msg)
74
span.add_event("update_key_dne", {"key": key})
75
raise KeyDoesNotExistError(msg)
76
77
def delete(key):
78
"""Delete key from the database."""
79
global db
80
81
with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
82
if key in db:
83
del db[key]
84
logging.debug("Successful delete")
85
span.set_attribute("key", key)
86
return True
87
88
return False
db.py

If you try to update a key that does not exist, your application adds a span event on the span.

Restart your simulator

Now that you've changed the application logic, you need to restart your simulator. Make sure you do this in the same terminal window where you set your environment variables.

Restart your simulator:

bash
$
python simulator.py

Now, your simulator is running again.

You've instrumented your application to send custom events with your spans. You've also restarted your simulator. Now, it's time to view your new data.

View your new data in New Relic

Step 1 of 6

Log into New Relic.

Step 2 of 6

Go back to your OpenTelemetry service:

Click database service

Notice that your errors have dropped to zero:

No errors

You won't be able to use error counts to know how many times your users try to read keys that aren't there or create keys that already are, so use the query builder to figure it out.

Tip

If you don't see a change in your data, compare your code to ours.

Step 3 of 6

Click Query your data:

Query your data

With our query builder, you can write arbitrary queries against our database.

Step 4 of 6

Click Query builder and enter the following query:

FROM SpanEvent SELECT count(*) FACET name

Query

This query counts all the span events from your account and groups them by their name.

Step 5 of 6

Click Run to see your data:

Run

Step 6 of 6

Change your chart type to Table for better readability:

Table view of your span events

Here, you see the number of times your application has seen each span event, organized into a nice table.

Tip

You can even add this table to a dashboard if you want to:

Add your chart to a dashboard

You've updated your app to stop reporting normal user actions as errors. At the same time, you've maintained the ability to query the number of occurences of these actions.

Summary

As the developer of speedeedeebee, you've now instrumented your application with OpenTelemetry to send manually-collected traces to New Relic. And because you've instrumented your app with OpenTelemetry instead of our Python agent, you have more flexibility in how you can use your data. For example, if you want to add additional backend data sources besides New Relic, you can easily change that without having to add another vendor-specific agent.

Homework

Now that you know how to instrument a Python application with OpenTelemetry and send that data to New Relic, here are some things you can do next to familiarize yourself even more with New Relic and OpenTelemetry:

Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.