Monday, March 18, 2019

Showing Seconds in Excel Date/Time Columns

This is a very simple solution, but clearly it has been a long time since I posted.

I was unable to post during the tenure at Nike. They are very obsessive about their brand, and as an employee, I was not allowed to post on my own blog.

As of today, I have been at Dish Network for 18 months. My new role is as a Senior Data Architect in the Big Data team. I will be supporting both the Big Data and Media Sales teams.

In this role, I am finally back to working directly with databases, this time Hadoop. Hadoop is completely new to me, so it's fun to learn.

Solution

Now to the solution, which is related to Excel, not Hadoop directly.

My Hadoop (Hive) query is pulling date/time columns with this:

SELECT ... from_unixtime(start_epoch) ...

It returns date/time values like this: "2018-12-14 17:05:15". But when I cut-and-paste that output into Excel, it drops the seconds from date/time values.

Here's how to show the seconds values in Excel.

  1. Select the data/time column
  2. Click the drop-down above "Number" in the ribbon (it defaults to "General")
  3. Select "More Number Formats..."
  4. Select "Custom"
  5. Under "Type:", enter "yyyy-mm-dd hh:mm:ss"
  6. Click OK


The date/time values now show like this: “2018-12-14 17:05:15”, exactly matching the output in Hive.

Tuesday, November 19, 2013

MongoDB and CAP

Just read this great post, Call me maybe: MongoDB and the wrap-up post, Call me maybe: final thoughts. They're part of a series called Jepsen on behavior of various NoSQL databases in failure scenarios.

Very geeky but very interesting.

Completed MongoDB M102

As I mentioned earlier, I signed up for "M102: MongoDB for DBAs". Great class.

I just completed it yesterday and got 100% on everything, homework and final.

If you're interested in learning about MongoDB, this is a great way to get a broad exposure in a limited amount of time. Because it includes weekly lectures and homework, you can't procrastinate like you can with that technical book on your desk that you planned to read through.

Speaking at Data Modeling Zone 2014

My boss, Ryan Smith, and I will co-present on modeling in NoSQL at Data Modeling Zone 2014 here in Portland.

It's not until October of next year, so we don't have things nailed down yet, but it will be fun to work with Ryan on this.

RESTful Service Best Practices

I was recently looking for reference material to back up my preference for storing date/time values in MongoDB in UTC using MongoDB's ISODate type. I found this great reference published at amazonaws.com:

RESTful Service Best Practices
Recommendations for Creating Web Services

It gives a solid recommendation for RESTful services:
"... always use the same format, including the time portion (along with timezone information) in the string. ISO 8601 time point format is a good solution, using the fully-enhanced format that includes hours, minutes, seconds and a decimal fraction of seconds (e.g. yyyy-MM-dd'T'HH:mm:ss.SSS'Z')."
It doesn't directly address storage directly, but it's such a great resource, I wanted to share it.

Wednesday, October 30, 2013

Exploring SQL Server 2012's sys.dm_exec_describe_first_result_set_for_object()

SQL Server 2012 introduces sys.dm_exec_describe_first_result_set_for_object(). The official documentation is here.

Let's work through some examples to discover what this new function delivers.

First, build a table that will be used to illustrate the function:
CREATE TABLE dbo.TestTable (
    
ColID       INT                     IDENTITY,
    
ColStr      CHAR(10)                NOT NULL,
    
ColMod              AS ColID % 10,
    
ColDef      bit                     NOT NULL    DEFAULT 0,
    
ColCalc0    INT                     NULL,
    
ColDec      DECIMAL(3,2)            NULL,
    
CONSTRAINT UIX_Test1_ColStr UNIQUE (ColStr)
);

Then build the simplest possible stored procedure that references that table:
CREATE PROCEDURE dbo.SimpleProc AS
SELECT
ColID FROM dbo.TestTable;

Now call the function:
SELECT * 
FROM sys.dm_exec_describe_first_result_set_for_object (
    
OBJECT_ID('dbo.SimpleProc'),
    
0

);

Remember that this stored procedures selects only one column. There are many columns in the result set. Here are some of the more interesting ones:
is_hidden:                0
column_ordinal:           1
is_nullable:              0
system_type_name:         INT
is_identity_column:       1
is_updateable:            0
is_computed_column:       0
ordinal_in_order_by_list: NULL
order_by_is_descending:   NULL
asdf






MongoDB Training

I am in the middle of taking a free on-line class: "M102: MongoDB for DBAs".

Very interesting and educational so far. Also a bit frustrating. As I don't have a background in JavaScript, there are certain parts of the class that I struggle with.

In general though, this is a great class and well worth your time if you are interested in MongoDB.

Wednesday, August 07, 2013

Just Do It!


My last day at Fiserv (second round) was Thursday, August 1st. On Monday, I started as an Expert Data Architect at Nike.

I am so excited about this change and really look forward to what I can learn and contribute at Nike.

Friday, June 28, 2013

Tuesday, June 25, 2013

SQL Server 2014 CTP1

CTP1 of SQL Server 2014 is now available! Go download it and start using it.

My first SQL 2014 article will be published at simple-talk soon. I will update this post with a link once the article is on-line.

Tuesday, March 26, 2013

My New Role at Fiserv: Patterns and Practices Manager

In addition to the Data Architect role that I will continue to play, I am now the Patterns and Practices Manager for the Digital Channels Enterprise Architecture team.

Here is the text right out of the March 12th e-mail that announced the change:
Rob will take on responsibility for managing our processes and best practices within the Group. He will continue to lead our Database design discipline, but will also take on management for the Corillian [Professional Services] Architects. Michael Hallabrin, Tim Sarna, and Rocco Martin will report in to Rob as part of the CO-L Solutions Architecture team.
I am really excited about this new role. Michael, Tim, and Rocco are a bunch of really smart guys, and I am enjoying working more closely with them.

Rob

Presenting on SQL Server Change Data Capture in Eugene

I will be speaking to the Eugene Area SQL Server User Group on CDC.

Topic: Deconstructing SQL Server's Change Data Capture


SQL Server's Change Data Capture feature was introduced with SQL 2008, but how much do you know about how it works? After attending this session, you will understand the various parts involved in CDC, know how to set it up, and know the relative performance effect.

Our focus here is not just showing how CDC works on the surface (there are many articles that cover that), but rather to see what we can find out about the various moving parts within SQL Server that actually make CDC function. Understanding the internals will help in an overall understanding of CDC.

Location/Time


Peace Health, 123 International Way
Springfield OR

April 9th at 11:30am

Monday, January 28, 2013

A Creative Use of Computed Columns

I love to see a creative solution to a technical problem.

How can I get that user out of my table quickly by JackLi

Great creative solution using indexed persisted computed columns.

The beauty of indexed persisted computed columns is that, yes, you have to pay the cost of doing the computation, but you only pay that cost at the point where you INSERT or UPDATE that particular column. If the data is mostly static as it is in this case, the one-time cost is very small.

Wednesday, January 16, 2013

You Can’t Sacrifice Partition Tolerance

Just finished reading this excellent article by Coda Hale:


I thought the bit about yield versus uptime was very interesting, as was the bit about harvest (harvest = data available/complete data). Historically, [the corporate] we have built systems that sacrifice yield in the event of a failure. I like the challenge/opportunity of building a system that reduces harvest but maintains yield.

Thursday, December 20, 2012

sys.dm_db_stats_properties() - Initial Results on SQL 2008 R2 SP2

My first sys.dm_db_stats_properties() post showed results for SQL Server 2012 SP1. Here, I will show how the results for filtered indexes on 2008 R2 SP2 are different.

The test script is exactly the same as before, just running on a different version.

MSDN Docs

The 2008 R2 version of the docs is here. Because the results were different, I pulled the text for both the 2012 document and the 2008 R2 document and compared them. There are no substantive differences.

Here are the only differences:
2008 R2 SP2"Requires that the user owns the table or the user is a member of the sysadmin fixed server role, the db_owner fixed database role, or the db_ddladmin fixed database role."
2012 SP1 adds the highlighted section"Requires that the user has select permissions on statistics columns or the user owns the table or the user is a member of the sysadmin fixed server role, the db_owner fixed database role, or the db_ddladmin fixed database role."

Results

Microsoft SQL Server 2008 R2 (SP2) - 10.50.4000.0 (X64) 
   Jun 28 2012 08:36:30 
   Copyright (c) Microsoft Corporation
   Enterprise Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1)

Beginning execution loop
Batch execution completed 10000 times.
name              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
----------------- ----------------- ------------------- ----- ------------ ----- --------------- --------------------

name              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
----------------- ----------------- ------------------- ----- ------------ ----- --------------- --------------------
AK_Test1_ColGuid  NULL              2012-12-20 09:10:09 10000 10000        132   10000           0
IX_Test1_Deleted  NULL              2012-12-20 09:10:09 10000 10000        1     10000           0
IX_Test1_StatusId NULL              2012-12-20 09:10:09 10000 10000        1     10000           0
PK_Test1_ColId    NULL              2012-12-20 09:10:09 10000 10000        3     10000           0

Update Count: 5000
name              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
----------------- ----------------- ------------------- ----- ------------ ----- --------------- --------------------
AK_Test1_ColGuid  NULL              2012-12-20 09:10:09 10000 10000        132   10000           0
IX_Test1_Deleted  NULL              2012-12-20 09:10:09 10000 10000        1     10000           5000
IX_Test1_StatusId NULL              2012-12-20 09:10:09 10000 10000        1     10000           5000
PK_Test1_ColId    NULL              2012-12-20 09:10:09 10000 10000        3     10000           0

Observations

Filtered Index Results
Filtered indexes are excluded from these results where they return NULLs in 2012.

The difference appears to be in the fact that the function returns a row in 2012 for a filtered index and does not return a row in 2008 R2 for a filtered index.


Wednesday, December 19, 2012

sys.dm_db_stats_properties() - Initial Results on SQL 2012 SP1

The SQL Server team just released a new feature, the DMF sys.dm_db_stats_properties(). This new feature is available in 2008 R2 SP2 and 2012 SP1.

These tests were run on SQL Server 2012 SP1 and will illustrate what the function returns in the normal case.

First, some references:
The test code below will
  1. create a new table with multiple indexes.
  2. insert 10,000 rows, then UPDATE STATISTICS.
  3. call sys.dm_db_stats_properties().
  4. update the StatusId and Deleted columns of 50% of the rows.
  5. call sys.dm_db_stats_properties().

Test Code


USE tempdb;GOPRINT @@VERSION;GOIF EXISTS (SELECT * FROM sys.tables WHERE OBJECT_ID('dbo.Test1') IS NOT NULL)
   DROP TABLE dbo.Test1;CREATE TABLE dbo.Test1 (
   ColId       INT                 IDENTITY,
   ColGuid     UNIQUEIDENTIFIER    NOT NULL DEFAULT NEWID(),
   StatusId    tinyint             NOT NULL DEFAULT 1,
   Deleted     bit                 NOT NULL DEFAULT 0,
   CONSTRAINT PK_Test1_ColId PRIMARY KEY CLUSTERED (ColId),
   CONSTRAINT AK_Test1_ColGuid UNIQUE (ColGuid)
);GOCREATE NONCLUSTERED INDEX IX_Test1_ColId_WhereDeletedEq1 ON dbo.Test1 (
   ColId) WHERE Deleted = 1;GOCREATE NONCLUSTERED INDEX IX_Test1_StatusId_WhereDeletedEq1 ON dbo.Test1 (
   StatusId) WHERE Deleted = 1;GOCREATE NONCLUSTERED INDEX IX_Test1_Deleted ON dbo.Test1 (
   Deleted);GOCREATE NONCLUSTERED INDEX IX_Test1_StatusId ON dbo.Test1 (
   StatusId);GO-- ==================== Initial insertINSERT INTO dbo.Test1 DEFAULT VALUES;GO 10000-- ==================== Call sys.dm_db_stats_properties()SELECT
   s.name,
   s.filter_definition,
   CONVERT(VARCHAR(19), p.last_updated, 120) AS last_updated,
   p.[rows],
   p.rows_sampled,
   p.steps,
   p.unfiltered_rows,
   p.modification_counterFROM sys.stats AS sCROSS APPLY sys.dm_db_stats_properties(s.[object_id], s.stats_id) AS pWHERE s.[object_id] = OBJECT_ID('dbo.Test1')ORDER BY s.has_filter, s.name;GO-- ==================== UPDATE STATISTICSUPDATE STATISTICS dbo.Test1;GO-- ==================== Call sys.dm_db_stats_properties()SELECT
   s.name,
   s.filter_definition,
   CONVERT(VARCHAR(19), p.last_updated, 120) AS last_updated,
   p.[rows],
   p.rows_sampled,
   p.steps,
   p.unfiltered_rows,
   p.modification_counterFROM sys.stats AS sCROSS APPLY sys.dm_db_stats_properties(s.[object_id], s.stats_id) AS pWHERE s.[object_id] = OBJECT_ID('dbo.Test1')ORDER BY s.has_filter, s.name;GO-- ==================== Update 50% of records (StatusId, Deleted)UPDATE dbo.Test1SET
   StatusId = 2,
   Deleted  = 1WHERE ColId % 2 = 0;PRINT 'Update Count: ' + CONVERT(VARCHAR(11), @@ROWCOUNT);GO-- ==================== Call sys.dm_db_stats_properties()SELECT
   s.name,
   s.filter_definition,
   CONVERT(VARCHAR(19), p.last_updated, 120) AS last_updated,
   p.[rows],
   p.rows_sampled,
   p.steps,
   p.unfiltered_rows,
   p.modification_counterFROM sys.stats AS sCROSS APPLY sys.dm_db_stats_properties(s.[object_id], s.stats_id) AS pWHERE s.[object_id] = OBJECT_ID('dbo.Test1')ORDER BY s.has_filter, s.name;GO

Results

Microsoft SQL Server 2012 (SP1) - 11.0.3000.0 (X64) 
   Oct 19 2012 13:38:57 
   Copyright (c) Microsoft Corporation
   Enterprise Edition (64-bit) on Windows NT 6.1  (Build 7601: Service Pack 1)

Beginning execution loop
Batch execution completed 10000 times.

name                              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
--------------------------------- ----------------- ------------------- ----  ------------ ----- --------------- --------------------
AK_Test1_ColGuid                  NULL              NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_Deleted                  NULL              NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_StatusId                 NULL              NULL                NULL  NULL         NULL  NULL            NULL
PK_Test1_ColId                    NULL              NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_ColId_WhereDeletedEq1    ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_StatusId_WhereDeletedEq1 ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL

name                              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
--------------------------------- ----------------- ------------------- ----- ------------ ----- --------------- --------------------
AK_Test1_ColGuid                  NULL              2012-12-19 14:59:41 10000 10000        136   10000           0
IX_Test1_Deleted                  NULL              2012-12-19 14:59:41 10000 10000        1     10000           0
IX_Test1_StatusId                 NULL              2012-12-19 14:59:41 10000 10000        1     10000           0
PK_Test1_ColId                    NULL              2012-12-19 14:59:41 10000 10000        3     10000           0
IX_Test1_ColId_WhereDeletedEq1    ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_StatusId_WhereDeletedEq1 ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL

Update Count: 5000

name                              filter_definition last_updated        rows  rows_sampled steps unfiltered_rows modification_counter
--------------------------------- ----------------- ------------------------- ------------ ----- --------------- --------------------
AK_Test1_ColGuid                  NULL              2012-12-19 14:59:41 10000 10000        136   10000           0
IX_Test1_Deleted                  NULL              2012-12-19 14:59:41 10000 10000        1     10000           5000
IX_Test1_StatusId                 NULL              2012-12-19 14:59:41 10000 10000        1     10000           5000
PK_Test1_ColId                    NULL              2012-12-19 14:59:41 10000 10000        3     10000           0
IX_Test1_ColId_WhereDeletedEq1    ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL
IX_Test1_StatusId_WhereDeletedEq1 ([Deleted]=(1))   NULL                NULL  NULL         NULL  NULL            NULL

Observations

NULLs
Not surprisingly, the result columns are all NULL before calling UPDATE STATISTICS.

Steps
Steps is an interesting statistic. Steps indicates the "number of steps in the histogram." Note that "The maximum number of steps is 200." (ref) You won't get 200 by simply adding more rows. Inserting 100,000 rows instead of 10,000 gives a steps value of 56 for the GUID index.

Filtered Indexes
I was never able to get filtered rows to return any statistics even though the docs say for the rows column, "If the statistics are filtered or correspond to a filtered index, the number of rows might be less than the number of rows in the table."

Is this a bug in SQL Server or in my query?

modification_counter
As expected, modification_counter is 0 before there are any modifications and after modifying 5,000 rows, the counts are 5,000 (except for the filtered indexes).

Monday, November 05, 2012

SQL Server Data Compression and TDE

I had been looking for a definitive statement on how TDE and compression work together. I found it here:

Data Compression: Strategy, Capacity Planning and Best Practices
TDE encrypts the pages when they are written to disk and decrypts them when they are read from disk into memory. Because data compression (as well as decompression) is performed on in-memory pages, data compression always sees unencrypted data, and hence the effectiveness and efficiency of data compression is not impacted by TDE.

Most other resources I had found only address how TDE messes up backup compression.

Thursday, August 23, 2012

SELECT INTO is faster than INSERT SELECT

I had heard that SELECT INTO is faster than INSERT SELECT, so I wanted to prove it.

Here are the results of testing in both FULL and SIMPLE recover models. Each test was run twice, once with INSERT-SELECT first and once with SELECT-INTO first.

The net result is that, at least for this test code on my hardware, using SELECT INTO is an order of magnitude faster in SIMPLE recover mode than INSERT SELECT. Using SELECT INTO is faster in FULL recover mode than INSERT SELECT, but the difference is less significant.

Test Results


Recovery Model: Simple
Order: INSERT-SELECT first
Elapsed Mcs - INSERT-SELECT: 8,391,839
Elapsed Mcs - SELECT-INTO: 962,096

Order: SELECT-INTO first
Elapsed Mcs - INSERT-SELECT:7,205,720
Elapsed Mcs - SELECT-INTO: 918,092

Recovery Model: Full
Order: INSERT-SELECT first
Elapsed Mcs - INSERT-SELECT:7,095,709
Elapsed Mcs - SELECT-INTO:1,289,129

Order: SELECT-INTO first
Elapsed Mcs - INSERT-SELECT: 6,274,628
Elapsed Mcs - SELECT-INTO:1,902,190

Test Script


USE MASTER;
GO
IF EXISTS (SELECT * FROM sys.databases WHERE name = 'SelectInto') ALTER DATABASE SelectInto SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
IF EXISTS (SELECT * FROM sys.databases WHERE name = 'SelectInto') DROP DATABASE  SelectInto;
GO
CREATE DATABASE SelectInto;
GO
ALTER DATABASE SelectInto SET RECOVERY FULL;
--ALTER DATABASE SelectInto SET RECOVERY SIMPLE;
GO
USE SelectInto;
GO

-- Source table: t1
CREATE TABLE dbo.t1 (
  
ColID       INT         NOT NULL,
  
Col1        CHAR(30)    NOT NULL DEFAULT 'asdf',
  
CONSTRAINT PK_t1 PRIMARY KEY CLUSTERED (ColID)
);

-- Populate t1
DECLARE @MaxRows    INT     = 1000000;
DECLARE @i          INT     = 0;
WHILE @i < @MaxRows
BEGIN
   SET
@i += 1;
  
INSERT INTO dbo.t1 (ColID) VALUES (@i);
END;

-- Destination heap: t2
CREATE TABLE dbo.t2 (
  
ColID       INT         NOT NULL,
  
Col1        CHAR(30)    NOT NULL DEFAULT 'asdf'
);

DECLARE @t1     time(7);
DECLARE @t2     time(7);

-- INSERT/SELECT
SET @t1 = SYSUTCDATETIME();

INSERT INTO dbo.t2 (ColID, Col1)
SELECT ColID, Col1 FROM dbo.t1;

SET @t2 = SYSUTCDATETIME();
SELECT DATEDIFF(mcs, @t1, @t2) AS ElapsedMcs;

-- SELECT INTO
SET @t1 = SYSUTCDATETIME();

SELECT ColID, Col1
INTO dbo.t3
FROM dbo.t1;

SET @t2 = SYSUTCDATETIME();
SELECT DATEDIFF(mcs, @t1, @t2) AS ElapsedMcs;

References


Monday, August 20, 2012

SQL Server CDC: Remember It's Not Free

Someone recently recommended that a developer use SQL Server's CDC (Change Data Capture) to create a history record for a transactional table because doing that work in a stored procedure "makes this stored procedure slow". Remember that there is no magic in CDC; it's not free.

Based on the research I did for this article:
Performance Testing SQL Server 2008's Change Data Capture functionality
"The average penalty for CDC with full recovery is 10.51%."

SQLskills' "IE1: Immersion Event on Internals and Performance"

I just attended the most intense and valuable week of training I have ever experienced: SQLskills' "IE1: Immersion Event on Internals and Performance". This is a full week of training given by Kimberly Tripp and Paul S. Randal.

The whole week was great. These are two of the best trainers in the business, period. In this class, they teach everything, so you're getting great content every minute of every day.

I highly recommend the class.