[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:04:17 dist-test-slave-dlb8 krb5kdc[28353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:04:17 dist-test-slave-dlb8 krb5kdc[28353](info): set up 2 sockets
May 02 14:04:17 dist-test-slave-dlb8 krb5kdc[28353](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240502 14:04:18.147346 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:04:18 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:04:18.330560 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46613
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:18.613135 28366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:18.613817 28366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:18.614426 28366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:18.643560 28366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:04:18.643923 28366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:18.644212 28366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:18.644438 28366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:04:18.644652 28366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:04:18.678577 28366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:18.679757 28366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:18.680711 28366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:18.695916 28372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:18.696637 28373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:19.786530 28374 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1090 milliseconds
W20240502 14:04:19.788069 28375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:19.794276 28366 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.101s user 0.352s sys 0.745s
W20240502 14:04:19.794627 28366 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.101s user 0.352s sys 0.745s
I20240502 14:04:19.794936 28366 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:04:19.796505 28366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:19.799861 28366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:19.801415 28366 hybrid_clock.cc:648] HybridClock initialized: now 1714658659801377 us; error 32 us; skew 500 ppm
I20240502 14:04:19.802592 28366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:19 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658659, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:19.811589 28366 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:04:19.817586 28366 webserver.cc:460] Webserver started at http://127.27.174.126:38779/ using document root <none> and password file <none>
I20240502 14:04:19.818889 28366 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:19.819175 28366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:19.819743 28366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:19.828454 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "61ed44dfa9a541fbbb537cf292101023"
format_stamp: "Formatted at 2024-05-02 14:04:19 on dist-test-slave-dlb8"
server_key: "dd9b5c0a220db2eca8e8b44bc1278f97"
server_key_iv: "e49b9a564fd9c00c2a2c8010fbc85403"
server_key_version: "encryptionkey@0"
I20240502 14:04:19.830183 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "61ed44dfa9a541fbbb537cf292101023"
format_stamp: "Formatted at 2024-05-02 14:04:19 on dist-test-slave-dlb8"
server_key: "dd9b5c0a220db2eca8e8b44bc1278f97"
server_key_iv: "e49b9a564fd9c00c2a2c8010fbc85403"
server_key_version: "encryptionkey@0"
I20240502 14:04:19.840194 28366 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20240502 14:04:19.848556 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:19.850024 28366 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20240502 14:04:19.850467 28366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "61ed44dfa9a541fbbb537cf292101023"
format_stamp: "Formatted at 2024-05-02 14:04:19 on dist-test-slave-dlb8"
server_key: "dd9b5c0a220db2eca8e8b44bc1278f97"
server_key_iv: "e49b9a564fd9c00c2a2c8010fbc85403"
server_key_version: "encryptionkey@0"
I20240502 14:04:19.850936 28366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:19.928512 28366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:19.933522 28366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:19.933954 28366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:19.997763 28366 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46613
I20240502 14:04:19.997829 28435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46613 every 8 connection(s)
I20240502 14:04:20.000444 28366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:04:20.005314 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:20.006547 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28366
I20240502 14:04:20.006955 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:04:20.009251 28345 external_mini_cluster.cc:1428] Setting key f7b17620082798c682c29e61eb0da5bd
I20240502 14:04:20.024526 28436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023: Bootstrap starting.
I20240502 14:04:20.029407 28436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:20.031260 28436 log.cc:824] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:20.037365 28436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023: No bootstrap required, opened a new log
May 02 14:04:20 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:20.061218 28436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } }
I20240502 14:04:20.061959 28436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:20.062278 28436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ed44dfa9a541fbbb537cf292101023, State: Initialized, Role: FOLLOWER
I20240502 14:04:20.063303 28436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } }
I20240502 14:04:20.064018 28436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:20.064407 28436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:20.064817 28436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:20.065712 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:20.012602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.27.174.126:46613)
0502 14:04:20.015356 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:20.015398 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:20.016528 (+ 1130us) server_negotiation.cc:407] Connection header received
0502 14:04:20.036678 (+ 20150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:20.036730 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:20.037201 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:20.037705 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:20.041521 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:20.043721 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:20.045200 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:20.045828 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:20.051469 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:20.051598 (+ 129us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:20.051647 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:20.051796 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:20.057765 (+ 5969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:20.059209 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:20.059243 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:20.059284 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:20.059534 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:20.060082 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:20.060112 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:20.060136 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:20.060912 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:20.061335 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:20.062207 (+ 872us) server_negotiation.cc:299] Negotiation successful
0502 14:04:20.062808 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":1622,"threads_started":1}
I20240502 14:04:20.070333 28436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } }
I20240502 14:04:20.071004 28436 leader_election.cc:302] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61ed44dfa9a541fbbb537cf292101023; no voters:
I20240502 14:04:20.072571 28436 leader_election.cc:288] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:20.072911 28442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:20.076216 28442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [term 1 LEADER]: Becoming Leader. State: Replica: 61ed44dfa9a541fbbb537cf292101023, State: Running, Role: LEADER
I20240502 14:04:20.077128 28436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:20.076947 28442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } }
I20240502 14:04:20.084479 28443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61ed44dfa9a541fbbb537cf292101023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } } }
I20240502 14:04:20.085239 28443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:20.086565 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61ed44dfa9a541fbbb537cf292101023. Latest consensus state: current_term: 1 leader_uuid: "61ed44dfa9a541fbbb537cf292101023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ed44dfa9a541fbbb537cf292101023" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46613 } } }
I20240502 14:04:20.087289 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:20.092895 28450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:20.104873 28450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:20.121258 28450 catalog_manager.cc:1347] Generated new cluster ID: bb0758647c564a0286542a4bd617abef
I20240502 14:04:20.121513 28450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:20.148617 28450 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:20.150048 28450 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:20.161074 28450 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 61ed44dfa9a541fbbb537cf292101023: Generated new TSK 0
I20240502 14:04:20.161911 28450 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:04:20.213936 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:20.510699 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:20.511214 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:20.512158 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:20.542552 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:20.542922 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:20.543855 28464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:04:20.579433 28464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:20.580536 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:20.581566 28464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:20.600757 28471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:20.603891 28470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:20.604626 28473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:20.603051 28464 server_base.cc:1034] running on GCE node
I20240502 14:04:21.694846 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:21.697722 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:21.699131 28464 hybrid_clock.cc:648] HybridClock initialized: now 1714658661699106 us; error 52 us; skew 500 ppm
I20240502 14:04:21.699935 28464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:21 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:21.705855 28464 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:04:21.711958 28464 webserver.cc:460] Webserver started at http://127.27.174.65:36421/ using document root <none> and password file <none>
I20240502 14:04:21.712827 28464 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:21.713022 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:21.713488 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:21.718581 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "a4bd65530e7842488ae1d5f104a6009f"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-dlb8"
server_key: "25c9d9739729dd95c2273ac33e27e8a7"
server_key_iv: "5963078d4c80aaa73cd163ea0b9cd5ac"
server_key_version: "encryptionkey@0"
I20240502 14:04:21.719692 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "a4bd65530e7842488ae1d5f104a6009f"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-dlb8"
server_key: "25c9d9739729dd95c2273ac33e27e8a7"
server_key_iv: "5963078d4c80aaa73cd163ea0b9cd5ac"
server_key_version: "encryptionkey@0"
I20240502 14:04:21.727377 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20240502 14:04:21.733247 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.734467 28464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:04:21.734833 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "a4bd65530e7842488ae1d5f104a6009f"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-dlb8"
server_key: "25c9d9739729dd95c2273ac33e27e8a7"
server_key_iv: "5963078d4c80aaa73cd163ea0b9cd5ac"
server_key_version: "encryptionkey@0"
I20240502 14:04:21.735318 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:21.794409 28464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:21.799726 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:21.800135 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:21.802493 28464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:21.806234 28464 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:21.806433 28464 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.806617 28464 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:21.806738 28464 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.933450 28464 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39117
I20240502 14:04:21.933552 28595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39117 every 8 connection(s)
I20240502 14:04:21.935989 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:04:21.938001 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28464
I20240502 14:04:21.938485 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:04:21.939921 28345 external_mini_cluster.cc:1428] Setting key 0fe3f359bd03f7bfe80d10e9140dc28d
May 02 14:04:21 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658661, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:04:21.969120 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:21.940537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42181 (local address 127.27.174.126:46613)
0502 14:04:21.942072 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:21.942096 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:21.943970 (+ 1874us) server_negotiation.cc:407] Connection header received
0502 14:04:21.948576 (+ 4606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:21.948612 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:21.948862 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:21.949038 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:21.951676 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:21.953103 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:21.954436 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:21.955035 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:21.958639 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:21.958734 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:21.958764 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:21.958926 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:21.962623 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:21.964024 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:21.964047 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:21.964060 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:21.964180 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:21.964852 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:21.964870 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:21.964882 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:21.965299 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:21.965522 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:21.966549 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0502 14:04:21.966981 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":776,"threads_started":1}
I20240502 14:04:21.969885 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:21.940850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46613 (local address 127.27.174.65:42181)
0502 14:04:21.943755 (+ 2905us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:21.943820 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:21.948196 (+ 4376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:21.949346 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:21.949385 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:21.950474 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:21.951387 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:21.951426 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:21.953346 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:21.953365 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:21.954188 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:21.954223 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:21.954525 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:21.955443 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:21.955545 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:21.958354 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:21.962839 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:21.962878 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:21.962930 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:21.963852 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:21.964337 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:21.964371 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:21.964402 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:21.964687 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:21.965514 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:21.965547 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:21.965940 (+ 393us) client_negotiation.cc:769] Sending connection context
0502 14:04:21.966314 (+ 374us) client_negotiation.cc:240] Negotiation successful
0502 14:04:21.966938 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"thread_start_us":1663,"threads_started":1}
I20240502 14:04:21.971843 28596 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46613
I20240502 14:04:21.972199 28596 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:21.973102 28596 heartbeater.cc:511] Master 127.27.174.126:46613 requested a full tablet report, sending...
I20240502 14:04:21.975965 28400 ts_manager.cc:194] Registered new tserver with Master: a4bd65530e7842488ae1d5f104a6009f (127.27.174.65:39117)
I20240502 14:04:21.978597 28400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:42181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:04:21.988044 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:22.287205 28602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:22.287662 28602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:22.288740 28602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:22.319224 28602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:22.319538 28602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:22.320466 28602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:04:22.354748 28602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:22.355953 28602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:22.356987 28602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:22.372397 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.981789 28596 heartbeater.cc:503] Master 127.27.174.126:46613 was elected leader, sending a full tablet report...
W20240502 14:04:22.372987 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:22.373322 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.374230 28602 server_base.cc:1034] running on GCE node
I20240502 14:04:23.435627 28602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:23.437928 28602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:23.439319 28602 hybrid_clock.cc:648] HybridClock initialized: now 1714658663439284 us; error 55 us; skew 500 ppm
I20240502 14:04:23.440069 28602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:23 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:23.451390 28602 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:04:23.456231 28602 webserver.cc:460] Webserver started at http://127.27.174.66:35225/ using document root <none> and password file <none>
I20240502 14:04:23.457111 28602 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:23.457288 28602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:23.457655 28602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:23.462494 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "ed8ade05dac04022810ac9deb9ae3b1c"
format_stamp: "Formatted at 2024-05-02 14:04:23 on dist-test-slave-dlb8"
server_key: "bba04d410eb39ce4a134109e21a90e98"
server_key_iv: "df51c50fa977e8841dbd47780e02f005"
server_key_version: "encryptionkey@0"
I20240502 14:04:23.463572 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8ade05dac04022810ac9deb9ae3b1c"
format_stamp: "Formatted at 2024-05-02 14:04:23 on dist-test-slave-dlb8"
server_key: "bba04d410eb39ce4a134109e21a90e98"
server_key_iv: "df51c50fa977e8841dbd47780e02f005"
server_key_version: "encryptionkey@0"
I20240502 14:04:23.470453 28602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:04:23.475701 28619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:23.476788 28602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:04:23.477140 28602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "ed8ade05dac04022810ac9deb9ae3b1c"
format_stamp: "Formatted at 2024-05-02 14:04:23 on dist-test-slave-dlb8"
server_key: "bba04d410eb39ce4a134109e21a90e98"
server_key_iv: "df51c50fa977e8841dbd47780e02f005"
server_key_version: "encryptionkey@0"
I20240502 14:04:23.477497 28602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:23.535315 28602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:23.540508 28602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:23.540946 28602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:23.543380 28602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:23.547220 28602 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:23.547420 28602 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:23.547664 28602 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:23.547819 28602 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:23.679131 28602 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:34889
I20240502 14:04:23.679234 28732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:34889 every 8 connection(s)
I20240502 14:04:23.681646 28602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:04:23.691638 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28602
I20240502 14:04:23.692039 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:04:23.692936 28345 external_mini_cluster.cc:1428] Setting key 918a676b2499b6ce8b1e3ab40b8324b2
May 02 14:04:23 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658663, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:04:23.715404 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:23.685701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:34867 (local address 127.27.174.126:46613)
0502 14:04:23.686899 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:23.686914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:23.689241 (+ 2327us) server_negotiation.cc:407] Connection header received
0502 14:04:23.693968 (+ 4727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:23.693999 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:23.694234 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:23.694389 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:23.697277 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:23.698687 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:23.700361 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:23.700998 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:23.705073 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:23.705146 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:23.705163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:23.705273 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:23.708955 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:23.710061 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:23.710098 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:23.710112 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:23.710243 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:23.710837 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:23.710858 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:23.710871 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:23.711237 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:23.711447 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:23.712411 (+ 964us) server_negotiation.cc:299] Negotiation successful
0502 14:04:23.712889 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":560,"threads_started":1}
I20240502 14:04:23.715833 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:23.686000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46613 (local address 127.27.174.66:34867)
0502 14:04:23.688998 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:23.689067 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:23.693597 (+ 4530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:23.694633 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:23.694659 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:23.695766 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:23.696980 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:23.697023 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:23.698930 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:23.698954 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:23.700100 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:23.700157 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:23.700486 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:23.701369 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:23.701444 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:23.704785 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:23.709165 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:23.709188 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:23.709217 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:23.709829 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:23.710424 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:23.710446 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:23.710461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:23.710658 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:23.711444 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:23.711474 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:23.711862 (+ 388us) client_negotiation.cc:769] Sending connection context
0502 14:04:23.712197 (+ 335us) client_negotiation.cc:240] Negotiation successful
0502 14:04:23.712805 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2295,"thread_start_us":1678,"threads_started":1}
I20240502 14:04:23.717844 28733 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46613
I20240502 14:04:23.718245 28733 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:23.719197 28733 heartbeater.cc:511] Master 127.27.174.126:46613 requested a full tablet report, sending...
I20240502 14:04:23.721300 28400 ts_manager.cc:194] Registered new tserver with Master: ed8ade05dac04022810ac9deb9ae3b1c (127.27.174.66:34889)
I20240502 14:04:23.722615 28400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:34867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:04:23.738832 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:24.051640 28739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:24.052155 28739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:24.052795 28739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:24.082141 28739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:24.082509 28739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:24.083407 28739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:04:24.117564 28739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:24.118721 28739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:24.119729 28739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:24.134893 28745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:24.725781 28733 heartbeater.cc:503] Master 127.27.174.126:46613 was elected leader, sending a full tablet report...
W20240502 14:04:24.135571 28746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:24.137645 28739 server_base.cc:1034] running on GCE node
W20240502 14:04:24.136514 28748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:25.215577 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:25.218411 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:25.219832 28739 hybrid_clock.cc:648] HybridClock initialized: now 1714658665219775 us; error 79 us; skew 500 ppm
I20240502 14:04:25.220582 28739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:25.226225 28739 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:04:25.231803 28739 webserver.cc:460] Webserver started at http://127.27.174.67:37059/ using document root <none> and password file <none>
I20240502 14:04:25.232687 28739 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:25.232884 28739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:25.233307 28739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:25.238533 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "9da25e2fb2ec4bfd84601172824b9c9d"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-dlb8"
server_key: "881102f41a9cfde52ddd700e7f0b46ad"
server_key_iv: "88d626a331f8fb9ae7971d68d09c8cad"
server_key_version: "encryptionkey@0"
I20240502 14:04:25.239713 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "9da25e2fb2ec4bfd84601172824b9c9d"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-dlb8"
server_key: "881102f41a9cfde52ddd700e7f0b46ad"
server_key_iv: "88d626a331f8fb9ae7971d68d09c8cad"
server_key_version: "encryptionkey@0"
I20240502 14:04:25.246717 28739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:04:25.252745 28756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.253755 28739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:04:25.254109 28739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "9da25e2fb2ec4bfd84601172824b9c9d"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-dlb8"
server_key: "881102f41a9cfde52ddd700e7f0b46ad"
server_key_iv: "88d626a331f8fb9ae7971d68d09c8cad"
server_key_version: "encryptionkey@0"
I20240502 14:04:25.254498 28739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:25.304536 28739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:25.310339 28739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:25.310798 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:25.313254 28739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:25.317250 28739 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:25.317462 28739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.317719 28739 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:25.317881 28739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.449258 28739 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:45019
I20240502 14:04:25.449333 28869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:45019 every 8 connection(s)
I20240502 14:04:25.452034 28739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:04:25.452858 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28739
I20240502 14:04:25.453244 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:04:25.454182 28345 external_mini_cluster.cc:1428] Setting key a23b28de30b6d7cf07f75a2455216c87
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:25.483160 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:25.457670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:58321 (local address 127.27.174.126:46613)
0502 14:04:25.458993 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:25.459015 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:25.461029 (+ 2014us) server_negotiation.cc:407] Connection header received
0502 14:04:25.465490 (+ 4461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:25.465511 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:25.465679 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:25.465822 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:25.468106 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.469030 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:25.470332 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.470746 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:25.474532 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:25.474603 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:25.474620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:25.474729 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:25.478405 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.479408 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.479428 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.479441 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.479563 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.480020 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.480036 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.480047 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.480407 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:25.480587 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:25.481406 (+ 819us) server_negotiation.cc:299] Negotiation successful
0502 14:04:25.481726 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":509,"threads_started":1}
I20240502 14:04:25.483691 28873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:25.457952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46613 (local address 127.27.174.67:58321)
0502 14:04:25.460833 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:25.460895 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:25.465087 (+ 4192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:25.466020 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:25.466074 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:25.466891 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:25.467806 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:25.467843 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.469230 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:25.469252 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:25.470087 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:25.470134 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.470423 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:25.471300 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:25.471361 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:25.474302 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:25.478603 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:25.478627 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:25.478649 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:25.479242 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:25.479686 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:25.479702 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:25.479715 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:25.479888 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:25.480570 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:25.480589 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:25.480938 (+ 349us) client_negotiation.cc:769] Sending connection context
0502 14:04:25.481226 (+ 288us) client_negotiation.cc:240] Negotiation successful
0502 14:04:25.481632 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2225,"thread_start_us":1639,"threads_started":1}
I20240502 14:04:25.485522 28870 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46613
I20240502 14:04:25.485940 28870 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:25.487180 28870 heartbeater.cc:511] Master 127.27.174.126:46613 requested a full tablet report, sending...
I20240502 14:04:25.489081 28399 ts_manager.cc:194] Registered new tserver with Master: 9da25e2fb2ec4bfd84601172824b9c9d (127.27.174.67:45019)
I20240502 14:04:25.490427 28399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:58321
I20240502 14:04:25.500741 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:25.561944 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:25.539088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.27.174.65:39117)
0502 14:04:25.541467 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:25.541502 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:25.541807 (+ 305us) server_negotiation.cc:407] Connection header received
0502 14:04:25.542201 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:25.542223 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:25.542475 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:25.542661 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:25.545204 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.546698 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:25.548115 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.548720 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:25.552781 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:25.552880 (+ 99us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:25.552906 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:25.552999 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:25.556750 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.557795 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.557818 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.557843 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.558003 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.558563 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.558582 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.558595 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.559013 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:25.559466 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:25.560048 (+ 582us) server_negotiation.cc:299] Negotiation successful
0502 14:04:25.560517 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1343,"threads_started":1}
W20240502 14:04:25.563367 28509 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57656
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:25.617869 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:25.597488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.27.174.65:39117)
0502 14:04:25.598101 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:25.598150 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:25.599941 (+ 1791us) server_negotiation.cc:407] Connection header received
0502 14:04:25.600244 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:25.600261 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:25.600432 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:25.600584 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:25.602905 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.603725 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:25.604959 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:25.605304 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:25.609582 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:25.609659 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:25.609678 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:25.609772 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:25.612959 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.613855 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.613876 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.613908 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.614035 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:25.614560 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:25.614576 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:25.614589 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:25.615021 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:25.615373 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:25.616322 (+ 949us) server_negotiation.cc:299] Negotiation successful
0502 14:04:25.616634 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20240502 14:04:25.620033 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28464
I20240502 14:04:25.643023 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28602
I20240502 14:04:25.667253 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28739
I20240502 14:04:25.689296 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28366
2024-05-02T14:04:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (8584 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28885](info): set up 2 sockets
May 02 14:04:25 dist-test-slave-dlb8 krb5kdc[28885](info): commencing operation
krb5kdc: starting...
W20240502 14:04:26.699139 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:04:26 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:04:26.890614 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35845
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:27.178829 28898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:27.179456 28898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:27.180024 28898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:27.210361 28898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:04:27.210717 28898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:27.211000 28898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:27.211241 28898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:04:27.211479 28898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:04:27.246760 28898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:27.247960 28898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:27.248991 28898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:27.264626 28906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:27.265183 28905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:27.265949 28898 server_base.cc:1034] running on GCE node
W20240502 14:04:27.265908 28908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:28.351399 28898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:28.354398 28898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:28.355840 28898 hybrid_clock.cc:648] HybridClock initialized: now 1714658668355813 us; error 47 us; skew 500 ppm
I20240502 14:04:28.356611 28898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:28 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658668, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:28.362454 28898 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:04:28.368008 28898 webserver.cc:460] Webserver started at http://127.27.174.126:43731/ using document root <none> and password file <none>
I20240502 14:04:28.368860 28898 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:28.369064 28898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:28.369499 28898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:28.374429 28898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "b97778b7ab624392ae0e2ad5ddb03da0"
format_stamp: "Formatted at 2024-05-02 14:04:28 on dist-test-slave-dlb8"
server_key: "d7126152521b6c09504852393edd4d8c"
server_key_iv: "195534b1d7e915b3f6730cceafd20e1a"
server_key_version: "encryptionkey@0"
I20240502 14:04:28.375527 28898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "b97778b7ab624392ae0e2ad5ddb03da0"
format_stamp: "Formatted at 2024-05-02 14:04:28 on dist-test-slave-dlb8"
server_key: "d7126152521b6c09504852393edd4d8c"
server_key_iv: "195534b1d7e915b3f6730cceafd20e1a"
server_key_version: "encryptionkey@0"
I20240502 14:04:28.382517 28898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:04:28.387837 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:28.388926 28898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:04:28.389273 28898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "b97778b7ab624392ae0e2ad5ddb03da0"
format_stamp: "Formatted at 2024-05-02 14:04:28 on dist-test-slave-dlb8"
server_key: "d7126152521b6c09504852393edd4d8c"
server_key_iv: "195534b1d7e915b3f6730cceafd20e1a"
server_key_version: "encryptionkey@0"
I20240502 14:04:28.389631 28898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:28.442749 28898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:28.448061 28898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:28.448496 28898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:28.514008 28898 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35845
I20240502 14:04:28.514122 28967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35845 every 8 connection(s)
I20240502 14:04:28.516727 28898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:04:28.521705 28968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:28.527294 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28898
I20240502 14:04:28.527783 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:04:28.528816 28345 external_mini_cluster.cc:1428] Setting key fd384b78783146237a62781314f767a6
I20240502 14:04:28.546458 28968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0: Bootstrap starting.
I20240502 14:04:28.551295 28968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:28.552925 28968 log.cc:824] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0: Log is configured to *not* fsync() on all Append() calls
May 02 14:04:28 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:28.558642 28968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0: No bootstrap required, opened a new log
I20240502 14:04:28.570693 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:28.531235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.27.174.126:35845)
0502 14:04:28.534174 (+ 2939us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:28.534218 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:28.534298 (+ 80us) server_negotiation.cc:407] Connection header received
0502 14:04:28.538690 (+ 4392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:28.538752 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:28.539254 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:28.539722 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:28.542378 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:28.544592 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:28.546088 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:28.546695 (+ 607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:28.557360 (+ 10665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:28.557490 (+ 130us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:28.557533 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:28.557685 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:28.563393 (+ 5708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:28.564613 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:28.564651 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:28.564696 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:28.564919 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:28.565413 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:28.565457 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:28.565481 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:28.566323 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:28.566785 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:28.567283 (+ 498us) server_negotiation.cc:299] Negotiation successful
0502 14:04:28.567871 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2273,"thread_start_us":1890,"threads_started":1}
I20240502 14:04:28.581640 28968 raft_consensus.cc:348] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } }
I20240502 14:04:28.582409 28968 raft_consensus.cc:374] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:28.582609 28968 raft_consensus.cc:729] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97778b7ab624392ae0e2ad5ddb03da0, State: Initialized, Role: FOLLOWER
I20240502 14:04:28.583215 28968 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } }
I20240502 14:04:28.583711 28968 raft_consensus.cc:388] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:28.583997 28968 raft_consensus.cc:482] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:28.584307 28968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:28.588236 28968 raft_consensus.cc:504] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } }
I20240502 14:04:28.588910 28968 leader_election.cc:302] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b97778b7ab624392ae0e2ad5ddb03da0; no voters:
I20240502 14:04:28.590498 28968 leader_election.cc:288] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:28.590852 28973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:28.593282 28973 raft_consensus.cc:686] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [term 1 LEADER]: Becoming Leader. State: Replica: b97778b7ab624392ae0e2ad5ddb03da0, State: Running, Role: LEADER
I20240502 14:04:28.594012 28973 consensus_queue.cc:237] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } }
I20240502 14:04:28.594897 28968 sys_catalog.cc:564] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:28.605880 28974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } } }
I20240502 14:04:28.606683 28974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:28.608289 28975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b97778b7ab624392ae0e2ad5ddb03da0. Latest consensus state: current_term: 1 leader_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97778b7ab624392ae0e2ad5ddb03da0" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35845 } } }
I20240502 14:04:28.610491 28975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:28.613903 28983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:28.626034 28983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:28.641644 28983 catalog_manager.cc:1347] Generated new cluster ID: 73d5ad533e5f4e46b04fb720511c80ce
I20240502 14:04:28.641980 28983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:28.655112 28983 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:28.656816 28983 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:28.672731 28983 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b97778b7ab624392ae0e2ad5ddb03da0: Generated new TSK 0
I20240502 14:04:28.673621 28983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:04:28.733105 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:29.047353 28994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:29.047899 28994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:29.048596 28994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:29.080564 28994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:29.080930 28994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:29.081899 28994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:04:29.117434 28994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:29.118634 28994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:29.119601 28994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:29.138120 29000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:29.140127 29003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:29.138450 29001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:30.256421 29002 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:04:30.256983 28994 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:04:30.260540 28994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:30.263401 28994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:30.264850 28994 hybrid_clock.cc:648] HybridClock initialized: now 1714658670264799 us; error 86 us; skew 500 ppm
I20240502 14:04:30.265868 28994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:30 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:30.271859 28994 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:04:30.277935 28994 webserver.cc:460] Webserver started at http://127.27.174.65:34137/ using document root <none> and password file <none>
I20240502 14:04:30.278997 28994 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:30.279204 28994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:30.279655 28994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:30.285022 28994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "6a2d59cb29a249b3a634003d4afb9e6e"
format_stamp: "Formatted at 2024-05-02 14:04:30 on dist-test-slave-dlb8"
server_key: "b177701194fc0c07ffb5effdbbfe1dbc"
server_key_iv: "86b0c3e79c1f7376f9a481c82a601027"
server_key_version: "encryptionkey@0"
I20240502 14:04:30.286269 28994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "6a2d59cb29a249b3a634003d4afb9e6e"
format_stamp: "Formatted at 2024-05-02 14:04:30 on dist-test-slave-dlb8"
server_key: "b177701194fc0c07ffb5effdbbfe1dbc"
server_key_iv: "86b0c3e79c1f7376f9a481c82a601027"
server_key_version: "encryptionkey@0"
I20240502 14:04:30.293921 28994 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240502 14:04:30.300185 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:30.301292 28994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20240502 14:04:30.301590 28994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "6a2d59cb29a249b3a634003d4afb9e6e"
format_stamp: "Formatted at 2024-05-02 14:04:30 on dist-test-slave-dlb8"
server_key: "b177701194fc0c07ffb5effdbbfe1dbc"
server_key_iv: "86b0c3e79c1f7376f9a481c82a601027"
server_key_version: "encryptionkey@0"
I20240502 14:04:30.301987 28994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:30.356302 28994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:30.361900 28994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:30.362385 28994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:30.364804 28994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:30.368731 28994 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:30.368921 28994 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:30.369175 28994 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:30.369323 28994 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:30.504534 28994 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35929
I20240502 14:04:30.504634 29124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35929 every 8 connection(s)
I20240502 14:04:30.507198 28994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:04:30.514505 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 28994
I20240502 14:04:30.515069 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:04:30.516716 28345 external_mini_cluster.cc:1428] Setting key 9b5d5a3bbed6262dd59fc5d791d43796
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
May 02 14:04:30 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658670, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
W20240502 14:04:30.808557 29128 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.266s user 0.003s sys 0.000s
I20240502 14:04:30.821003 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:30.511303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:56887 (local address 127.27.174.126:35845)
0502 14:04:30.515185 (+ 3882us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:30.515210 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:30.530173 (+ 14963us) server_negotiation.cc:407] Connection header received
0502 14:04:30.533052 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:30.533082 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:30.533321 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:30.533520 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:30.536954 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:30.538306 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:30.540394 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:30.540752 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:30.809501 (+268749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:30.809624 (+ 123us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:30.809678 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:30.809865 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:30.814471 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:30.816231 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:30.816256 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:30.816269 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:30.816401 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:30.817146 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:30.817167 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:30.817180 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:30.817573 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:30.817748 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:30.819161 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0502 14:04:30.819593 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":2051,"threads_started":1}
I20240502 14:04:30.823216 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:30.511616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35845 (local address 127.27.174.65:56887)
0502 14:04:30.526326 (+ 14710us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:30.526432 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:30.532621 (+ 6189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:30.533834 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:30.533884 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:30.535185 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:30.536613 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:30.536685 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:30.538593 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:30.538634 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:30.540109 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:30.540180 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:30.540669 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:30.541848 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:30.541955 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:30.809040 (+267085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:30.814739 (+ 5699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:30.814788 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:30.814837 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:30.816018 (+ 1181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:30.816580 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:30.816618 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:30.816659 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:30.816964 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:30.817799 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:30.817853 (+ 54us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:30.818489 (+ 636us) client_negotiation.cc:769] Sending connection context
0502 14:04:30.818978 (+ 489us) client_negotiation.cc:240] Negotiation successful
0502 14:04:30.819669 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13646,"thread_start_us":18152,"threads_started":1}
I20240502 14:04:30.825223 29125 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35845
I20240502 14:04:30.825590 29125 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:30.826586 29125 heartbeater.cc:511] Master 127.27.174.126:35845 requested a full tablet report, sending...
I20240502 14:04:30.829063 28933 ts_manager.cc:194] Registered new tserver with Master: 6a2d59cb29a249b3a634003d4afb9e6e (127.27.174.65:35929)
I20240502 14:04:30.831002 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:56887
I20240502 14:04:31.834252 29125 heartbeater.cc:503] Master 127.27.174.126:35845 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
W20240502 14:04:31.959411 28345 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.174.66: real 1.436s user 0.004s sys 0.009s
I20240502 14:04:31.961059 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:32.287505 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:32.287999 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:32.288609 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:32.322247 29131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:32.322643 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:32.323601 29131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:04:32.361382 29131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:32.362658 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:32.363771 29131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:32.379756 29138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:32.380489 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:32.381778 29131 server_base.cc:1034] running on GCE node
W20240502 14:04:32.381475 29140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:33.484970 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:33.487843 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:33.489302 29131 hybrid_clock.cc:648] HybridClock initialized: now 1714658673489252 us; error 70 us; skew 500 ppm
I20240502 14:04:33.490144 29131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:33 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658673, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:33.496412 29131 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:04:33.501395 29131 webserver.cc:460] Webserver started at http://127.27.174.66:38097/ using document root <none> and password file <none>
I20240502 14:04:33.502358 29131 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:33.502552 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:33.503002 29131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:33.508059 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c63db1ffb63d47148f06e551849371f6"
format_stamp: "Formatted at 2024-05-02 14:04:33 on dist-test-slave-dlb8"
server_key: "d70760c655d167f29731249876af55a2"
server_key_iv: "8e3f31b0b780e2b2dd2eb320ec85d3ff"
server_key_version: "encryptionkey@0"
I20240502 14:04:33.509264 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c63db1ffb63d47148f06e551849371f6"
format_stamp: "Formatted at 2024-05-02 14:04:33 on dist-test-slave-dlb8"
server_key: "d70760c655d167f29731249876af55a2"
server_key_iv: "8e3f31b0b780e2b2dd2eb320ec85d3ff"
server_key_version: "encryptionkey@0"
I20240502 14:04:33.517233 29131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20240502 14:04:33.523218 29148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:33.524379 29131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:04:33.524703 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "c63db1ffb63d47148f06e551849371f6"
format_stamp: "Formatted at 2024-05-02 14:04:33 on dist-test-slave-dlb8"
server_key: "d70760c655d167f29731249876af55a2"
server_key_iv: "8e3f31b0b780e2b2dd2eb320ec85d3ff"
server_key_version: "encryptionkey@0"
I20240502 14:04:33.525041 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:33.581387 29131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:33.586630 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:33.587095 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:33.589555 29131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:33.593390 29131 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:33.593580 29131 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:33.593844 29131 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:33.593986 29131 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:33.725033 29131 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:40453
I20240502 14:04:33.725143 29261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:40453 every 8 connection(s)
I20240502 14:04:33.727628 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:04:33.737015 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29131
I20240502 14:04:33.737540 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:04:33.738873 28345 external_mini_cluster.cc:1428] Setting key fd2d4aec7ffb4dd8bd1b0eb25c857f88
May 02 14:04:33 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658673, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:04:33.763772 29264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:33.731651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53527 (local address 127.27.174.126:35845)
0502 14:04:33.732729 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:33.732744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:33.735146 (+ 2402us) server_negotiation.cc:407] Connection header received
0502 14:04:33.741337 (+ 6191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:33.741362 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:33.741589 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:33.741796 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:33.744677 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:33.745539 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:33.747593 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:33.748111 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:33.752390 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:33.752513 (+ 123us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:33.752551 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:33.752718 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:33.758128 (+ 5410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:33.759316 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:33.759363 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:33.759404 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:33.759610 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:33.760200 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:33.760236 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:33.760257 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:33.760831 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:33.761085 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:33.761998 (+ 913us) server_negotiation.cc:299] Negotiation successful
0502 14:04:33.762422 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":441,"threads_started":1}
I20240502 14:04:33.764434 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:33.731977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35845 (local address 127.27.174.66:53527)
0502 14:04:33.734921 (+ 2944us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:33.734983 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:33.740947 (+ 5964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:33.741984 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:33.742009 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:33.743025 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:33.744378 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:33.744441 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:33.745813 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:33.745854 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:33.747305 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:33.747367 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:33.747833 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:33.748862 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:33.748931 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:33.752067 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:33.758380 (+ 6313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:33.758404 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:33.758430 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:33.759074 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:33.759787 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:33.759805 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:33.759819 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:33.760004 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:33.761012 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:33.761031 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:33.761387 (+ 356us) client_negotiation.cc:769] Sending connection context
0502 14:04:33.761708 (+ 321us) client_negotiation.cc:240] Negotiation successful
0502 14:04:33.762235 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2264,"thread_start_us":1660,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:04:33.766489 29262 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35845
I20240502 14:04:33.766971 29262 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:33.768360 29262 heartbeater.cc:511] Master 127.27.174.126:35845 requested a full tablet report, sending...
I20240502 14:04:33.770869 28933 ts_manager.cc:194] Registered new tserver with Master: c63db1ffb63d47148f06e551849371f6 (127.27.174.66:40453)
I20240502 14:04:33.772526 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:53527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:04:33.790118 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:34.089291 29268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:34.089872 29268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:34.090556 29268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:34.120378 29268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:34.120747 29268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:34.121721 29268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:04:34.156322 29268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:34.157565 29268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:34.158617 29268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:34.176864 29274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:34.775512 29262 heartbeater.cc:503] Master 127.27.174.126:35845 was elected leader, sending a full tablet report...
W20240502 14:04:34.177913 29275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:34.178582 29268 server_base.cc:1034] running on GCE node
W20240502 14:04:34.177671 29277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.238735 29268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:35.241529 29268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:35.243005 29268 hybrid_clock.cc:648] HybridClock initialized: now 1714658675242952 us; error 94 us; skew 500 ppm
I20240502 14:04:35.243777 29268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:35.249490 29268 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:04:35.254457 29268 webserver.cc:460] Webserver started at http://127.27.174.67:44691/ using document root <none> and password file <none>
I20240502 14:04:35.255328 29268 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:35.255518 29268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:35.255949 29268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:35.260960 29268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "53367879da3f498b826956b64b8db2ce"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-dlb8"
server_key: "6ba41721ffcb3e292f4c158a73bae683"
server_key_iv: "c7b349a523d6c635c307c16c6ecfa528"
server_key_version: "encryptionkey@0"
I20240502 14:04:35.262158 29268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "53367879da3f498b826956b64b8db2ce"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-dlb8"
server_key: "6ba41721ffcb3e292f4c158a73bae683"
server_key_iv: "c7b349a523d6c635c307c16c6ecfa528"
server_key_version: "encryptionkey@0"
I20240502 14:04:35.268987 29268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:04:35.274338 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.275360 29268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:04:35.275689 29268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "53367879da3f498b826956b64b8db2ce"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-dlb8"
server_key: "6ba41721ffcb3e292f4c158a73bae683"
server_key_iv: "c7b349a523d6c635c307c16c6ecfa528"
server_key_version: "encryptionkey@0"
I20240502 14:04:35.276050 29268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:35.334448 29268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:35.340103 29268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:35.340610 29268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:35.343160 29268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:35.347095 29268 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:35.347316 29268 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.347555 29268 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:35.347733 29268 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.474066 29268 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41567
I20240502 14:04:35.474241 29397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41567 every 8 connection(s)
I20240502 14:04:35.476438 29268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:04:35.480334 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29268
I20240502 14:04:35.480792 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:04:35.482153 28345 external_mini_cluster.cc:1428] Setting key 418e3d0bd5e1140305663fa05990cca9
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:35.507644 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.480450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:46305 (local address 127.27.174.126:35845)
0502 14:04:35.482171 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.482195 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.483493 (+ 1298us) server_negotiation.cc:407] Connection header received
0502 14:04:35.488474 (+ 4981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.488493 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.488653 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.488788 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:35.491002 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.491927 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.493136 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.493451 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.497138 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:35.497219 (+ 81us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:35.497237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:35.497333 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:35.501414 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.502515 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.502540 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.502553 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.502684 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.503309 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.503329 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.503342 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.503716 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:35.503932 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.504856 (+ 924us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.505300 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":979,"threads_started":1}
I20240502 14:04:35.508312 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.480794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35845 (local address 127.27.174.67:46305)
0502 14:04:35.483256 (+ 2462us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.483322 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.488133 (+ 4811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.489002 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.489026 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.489769 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:35.490783 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.490821 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.492129 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.492147 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.492969 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.492999 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.493267 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.494027 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:35.494134 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:35.496892 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:35.501594 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:35.501616 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:35.501656 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:35.502312 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:35.502850 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:35.502872 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:35.502887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:35.503128 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:35.503913 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:35.503944 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:35.504326 (+ 382us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.504650 (+ 324us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.505252 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":1485,"threads_started":1}
I20240502 14:04:35.510076 29398 heartbeater.cc:347] Connected to a master server at 127.27.174.126:35845
I20240502 14:04:35.510526 29398 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:35.511790 29398 heartbeater.cc:511] Master 127.27.174.126:35845 requested a full tablet report, sending...
I20240502 14:04:35.515187 28933 ts_manager.cc:194] Registered new tserver with Master: 53367879da3f498b826956b64b8db2ce (127.27.174.67:41567)
I20240502 14:04:35.516794 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:46305
I20240502 14:04:35.528254 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:04:35.564291 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.545745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.27.174.126:35845)
0502 14:04:35.546412 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.546449 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.547722 (+ 1273us) server_negotiation.cc:407] Connection header received
0502 14:04:35.548066 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.548084 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.548253 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.548393 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:35.550813 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.551609 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.552938 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.553470 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.555772 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:35.555844 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:35.555861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:35.555957 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:35.559400 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.560386 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.560412 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.560426 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.560564 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.561093 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.561110 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.561122 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.561497 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:35.561858 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.562509 (+ 651us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.562983 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20240502 14:04:35.572069 28933 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:35.591547 28933 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:35.635957 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.617372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.27.174.66:40453)
0502 14:04:35.620029 (+ 2657us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.620063 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.623810 (+ 3747us) server_negotiation.cc:407] Connection header received
0502 14:04:35.624530 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.624563 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.624929 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.625108 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:35.627123 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.629388 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.631877 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.633137 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.633405 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.634192 (+ 787us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.634446 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1987,"thread_start_us":3388,"threads_started":1}
I20240502 14:04:35.635999 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.619509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35929 (local address 127.0.0.1:58468)
0502 14:04:35.625526 (+ 6017us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.625776 (+ 250us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.626011 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.627576 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.627610 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.627720 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:35.628637 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.628688 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.631468 (+ 2780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.631498 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.633286 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.633335 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.633683 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.633803 (+ 120us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.633983 (+ 180us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.634255 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4644,"thread_start_us":2523,"threads_started":1}
I20240502 14:04:35.638502 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.616956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:40453 (local address 127.0.0.1:45656)
0502 14:04:35.623551 (+ 6595us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.623640 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.624145 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.625668 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.625696 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.625809 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:35.626794 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.626848 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.629746 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.629775 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.631591 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.631637 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.633683 (+ 2046us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.633803 (+ 120us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.636615 (+ 2812us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.636856 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5502,"thread_start_us":6900,"threads_started":1}
I20240502 14:04:35.640154 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.619797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.27.174.65:35929)
0502 14:04:35.624535 (+ 4738us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.624569 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.626258 (+ 1689us) server_negotiation.cc:407] Connection header received
0502 14:04:35.626493 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.626529 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.626874 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.627089 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:35.628959 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.631154 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.636582 (+ 5428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.637784 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.638002 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.638226 (+ 224us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.638492 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3895,"thread_start_us":3463,"threads_started":1}
I20240502 14:04:35.643359 29196 tablet_service.cc:1467] Processing CreateTablet for tablet 460c2f826f014737b93fdf02a4e4bfe6 (DEFAULT_TABLE table=test-table [id=952c5b59c7ad422588f95333f5142920]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:35.644840 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460c2f826f014737b93fdf02a4e4bfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:35.644528 29060 tablet_service.cc:1467] Processing CreateTablet for tablet 460c2f826f014737b93fdf02a4e4bfe6 (DEFAULT_TABLE table=test-table [id=952c5b59c7ad422588f95333f5142920]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:35.646430 29060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460c2f826f014737b93fdf02a4e4bfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:35.647611 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.624760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41567 (local address 127.0.0.1:56662)
0502 14:04:35.632888 (+ 8128us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.632945 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.633182 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.638495 (+ 5313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.638527 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.638646 (+ 119us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:35.639591 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.639645 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.642836 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.642869 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.644861 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.644919 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.645264 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.645358 (+ 94us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.645505 (+ 147us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.645722 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7346,"thread_start_us":8539,"threads_started":1}
I20240502 14:04:35.650770 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.625686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.27.174.67:41567)
0502 14:04:35.637420 (+ 11734us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.637461 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.637517 (+ 56us) server_negotiation.cc:407] Connection header received
0502 14:04:35.637709 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.637750 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.638134 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.638360 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:35.639982 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.642506 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.645235 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.646783 (+ 1548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.647048 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.648725 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.648993 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11053,"thread_start_us":10467,"threads_started":1}
I20240502 14:04:35.655054 29333 tablet_service.cc:1467] Processing CreateTablet for tablet 460c2f826f014737b93fdf02a4e4bfe6 (DEFAULT_TABLE table=test-table [id=952c5b59c7ad422588f95333f5142920]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:35.658524 29333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460c2f826f014737b93fdf02a4e4bfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:35.670701 29417 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Bootstrap starting.
I20240502 14:04:35.672888 29418 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Bootstrap starting.
I20240502 14:04:35.677996 29417 tablet_bootstrap.cc:654] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.679908 29418 tablet_bootstrap.cc:654] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.682271 29418 log.cc:824] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:35.682271 29417 log.cc:824] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:35.684604 29419 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Bootstrap starting.
I20240502 14:04:35.689139 29418 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: No bootstrap required, opened a new log
I20240502 14:04:35.689554 29418 ts_tablet_manager.cc:1367] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20240502 14:04:35.690114 29417 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: No bootstrap required, opened a new log
I20240502 14:04:35.690601 29417 ts_tablet_manager.cc:1367] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20240502 14:04:35.691903 29419 tablet_bootstrap.cc:654] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.694223 29419 log.cc:824] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:35.700846 29419 tablet_bootstrap.cc:492] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: No bootstrap required, opened a new log
I20240502 14:04:35.701362 29419 ts_tablet_manager.cc:1367] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20240502 14:04:35.712407 29418 raft_consensus.cc:348] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.713398 29418 raft_consensus.cc:374] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.713701 29418 raft_consensus.cc:729] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2d59cb29a249b3a634003d4afb9e6e, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.714917 29418 consensus_queue.cc:260] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.718663 29418 ts_tablet_manager.cc:1398] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Time spent starting tablet: real 0.029s user 0.023s sys 0.008s
I20240502 14:04:35.720264 29417 raft_consensus.cc:348] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.721091 29417 raft_consensus.cc:374] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.721388 29417 raft_consensus.cc:729] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63db1ffb63d47148f06e551849371f6, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.722307 29417 consensus_queue.cc:260] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.731601 29417 ts_tablet_manager.cc:1398] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Time spent starting tablet: real 0.041s user 0.034s sys 0.005s
W20240502 14:04:35.734124 29263 tablet.cc:2369] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:35.734932 29419 raft_consensus.cc:348] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.735878 29419 raft_consensus.cc:374] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.736176 29419 raft_consensus.cc:729] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53367879da3f498b826956b64b8db2ce, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.737074 29419 consensus_queue.cc:260] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.740331 29398 heartbeater.cc:503] Master 127.27.174.126:35845 was elected leader, sending a full tablet report...
I20240502 14:04:35.741055 29419 ts_tablet_manager.cc:1398] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Time spent starting tablet: real 0.039s user 0.037s sys 0.000s
W20240502 14:04:35.775372 29126 tablet.cc:2369] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:35.821835 29425 raft_consensus.cc:482] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:35.822384 29425 raft_consensus.cc:504] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.824713 29425 leader_election.cc:288] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63db1ffb63d47148f06e551849371f6 (127.27.174.66:40453), 6a2d59cb29a249b3a634003d4afb9e6e (127.27.174.65:35929)
I20240502 14:04:35.833897 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.825165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:40453 (local address 127.27.174.67:59881)
0502 14:04:35.827201 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.827245 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.827515 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.828299 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.828326 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.828421 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:35.828912 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.828969 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.830368 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.830390 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.831652 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.831706 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.832037 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.832128 (+ 91us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.832260 (+ 132us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.832457 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20240502 14:04:35.834272 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.825361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:59881 (local address 127.27.174.66:40453)
0502 14:04:35.826124 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.826150 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.827376 (+ 1226us) server_negotiation.cc:407] Connection header received
0502 14:04:35.827822 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.827850 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.828093 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.828230 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:35.829173 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.830186 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.831917 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.832624 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.832719 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.832846 (+ 127us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.832991 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20240502 14:04:35.835172 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.825232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35929 (local address 127.27.174.67:53451)
0502 14:04:35.827359 (+ 2127us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:35.827413 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:35.827783 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:35.828576 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:35.828602 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:35.828703 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:35.829641 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.829677 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.831036 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.831066 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:35.832818 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:35.832865 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.833173 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.833260 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:04:35.833395 (+ 135us) client_negotiation.cc:240] Negotiation successful
0502 14:04:35.833568 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1537,"thread_start_us":1232,"threads_started":1}
I20240502 14:04:35.835966 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.825927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:53451 (local address 127.27.174.65:35929)
0502 14:04:35.827136 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.827164 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.827539 (+ 375us) server_negotiation.cc:407] Connection header received
0502 14:04:35.828089 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.828116 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.828371 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.828520 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:35.829870 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.830803 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.833097 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.834311 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.834449 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.834632 (+ 183us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.834840 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20240502 14:04:35.835909 29217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460c2f826f014737b93fdf02a4e4bfe6" candidate_uuid: "53367879da3f498b826956b64b8db2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63db1ffb63d47148f06e551849371f6" is_pre_election: true
I20240502 14:04:35.836712 29217 raft_consensus.cc:2454] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53367879da3f498b826956b64b8db2ce in term 0.
I20240502 14:04:35.837330 29080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460c2f826f014737b93fdf02a4e4bfe6" candidate_uuid: "53367879da3f498b826956b64b8db2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" is_pre_election: true
I20240502 14:04:35.837868 29287 leader_election.cc:302] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53367879da3f498b826956b64b8db2ce, c63db1ffb63d47148f06e551849371f6; no voters:
I20240502 14:04:35.838188 29080 raft_consensus.cc:2454] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53367879da3f498b826956b64b8db2ce in term 0.
I20240502 14:04:35.838593 29425 raft_consensus.cc:2781] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:35.838922 29425 raft_consensus.cc:482] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:35.839268 29425 raft_consensus.cc:3037] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.843923 29425 raft_consensus.cc:504] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.845268 29425 leader_election.cc:288] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [CANDIDATE]: Term 1 election: Requested vote from peers c63db1ffb63d47148f06e551849371f6 (127.27.174.66:40453), 6a2d59cb29a249b3a634003d4afb9e6e (127.27.174.65:35929)
I20240502 14:04:35.845988 29217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460c2f826f014737b93fdf02a4e4bfe6" candidate_uuid: "53367879da3f498b826956b64b8db2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63db1ffb63d47148f06e551849371f6"
I20240502 14:04:35.846328 29080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460c2f826f014737b93fdf02a4e4bfe6" candidate_uuid: "53367879da3f498b826956b64b8db2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2d59cb29a249b3a634003d4afb9e6e"
I20240502 14:04:35.846582 29217 raft_consensus.cc:3037] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.846875 29080 raft_consensus.cc:3037] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.853603 29080 raft_consensus.cc:2454] T 460c2f826f014737b93fdf02a4e4bfe6 P 6a2d59cb29a249b3a634003d4afb9e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53367879da3f498b826956b64b8db2ce in term 1.
I20240502 14:04:35.853603 29217 raft_consensus.cc:2454] T 460c2f826f014737b93fdf02a4e4bfe6 P c63db1ffb63d47148f06e551849371f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53367879da3f498b826956b64b8db2ce in term 1.
I20240502 14:04:35.854620 29287 leader_election.cc:302] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53367879da3f498b826956b64b8db2ce, c63db1ffb63d47148f06e551849371f6; no voters:
I20240502 14:04:35.855226 29425 raft_consensus.cc:2781] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:35.856761 29425 raft_consensus.cc:686] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [term 1 LEADER]: Becoming Leader. State: Replica: 53367879da3f498b826956b64b8db2ce, State: Running, Role: LEADER
I20240502 14:04:35.857507 29425 consensus_queue.cc:237] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } }
I20240502 14:04:35.868069 28932 catalog_manager.cc:5484] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 53367879da3f498b826956b64b8db2ce (127.27.174.67). New cstate: current_term: 1 leader_uuid: "53367879da3f498b826956b64b8db2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63db1ffb63d47148f06e551849371f6" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53367879da3f498b826956b64b8db2ce" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a2d59cb29a249b3a634003d4afb9e6e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35929 } health_report { overall_health: UNKNOWN } } }
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:35.900872 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.880207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.27.174.65:35929)
0502 14:04:35.880743 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.880777 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.882085 (+ 1308us) server_negotiation.cc:407] Connection header received
0502 14:04:35.882402 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.882421 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.882600 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.882749 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:35.885066 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.885831 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.887227 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.887789 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.891900 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:35.891981 (+ 81us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:35.892014 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:35.892109 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:35.895858 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.896874 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.896897 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.896919 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.897074 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.897607 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.897627 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.897643 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.898128 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:35.898569 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.899105 (+ 536us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.899547 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:35.959903 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.940083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.27.174.65:35929)
0502 14:04:35.940742 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.940782 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.942568 (+ 1786us) server_negotiation.cc:407] Connection header received
0502 14:04:35.942941 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:35.942958 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:35.943113 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:35.943258 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:35.945403 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.946178 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:35.947471 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:35.948004 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:35.952122 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:35.952204 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:35.952221 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:35.952316 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:35.955287 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.956227 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.956247 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.956272 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.956397 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:35.957028 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:35.957046 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:35.957060 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:35.957423 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:35.957691 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:35.958293 (+ 602us) server_negotiation.cc:299] Negotiation successful
0502 14:04:35.958595 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20240502 14:04:35.981376 29399 tablet.cc:2369] T 460c2f826f014737b93fdf02a4e4bfe6 P 53367879da3f498b826956b64b8db2ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 02 14:04:35 dist-test-slave-dlb8 krb5kdc[28885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 02 14:04:36 dist-test-slave-dlb8 krb5kdc[28885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:36.017946 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:35.997765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.27.174.65:35929)
0502 14:04:35.998656 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:35.998690 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:35.999783 (+ 1093us) server_negotiation.cc:407] Connection header received
0502 14:04:36.000071 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:36.000088 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:36.000274 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:36.000417 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:36.002663 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:36.003442 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:36.004625 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:36.004957 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:36.009350 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:36.009423 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:36.009443 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:36.009547 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:36.012930 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:36.014033 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:36.014087 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:36.014106 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:36.014264 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:36.014803 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:36.014822 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:36.014838 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:36.015188 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:36.015523 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:36.016085 (+ 562us) server_negotiation.cc:299] Negotiation successful
0502 14:04:36.016550 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20240502 14:04:36.020402 29040 tablet_service.cc:2908] Rejecting scan request for tablet 460c2f826f014737b93fdf02a4e4bfe6: Uninitialized: clean time has not yet been initialized
I20240502 14:04:36.023900 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28994
I20240502 14:04:36.048396 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29131
I20240502 14:04:36.073393 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29268
I20240502 14:04:36.096256 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 28898
2024-05-02T14:04:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (10408 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:04:36 dist-test-slave-dlb8 krb5kdc[29441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:04:36 dist-test-slave-dlb8 krb5kdc[29441](info): set up 2 sockets
May 02 14:04:36 dist-test-slave-dlb8 krb5kdc[29441](info): commencing operation
krb5kdc: starting...
W20240502 14:04:37.103276 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:04:37 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:04:37.302585 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38089
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:37.589320 29455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:37.589906 29455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:37.590513 29455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:37.619727 29455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:04:37.620033 29455 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240502 14:04:37.620277 29455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:37.620548 29455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:37.620770 29455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:04:37.620987 29455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:04:37.655509 29455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:38089
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:37.656678 29455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:37.657676 29455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:37.672219 29462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:37.673143 29461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:37.674829 29455 server_base.cc:1034] running on GCE node
W20240502 14:04:37.673465 29464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:38.738588 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:38.741164 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:38.742501 29455 hybrid_clock.cc:648] HybridClock initialized: now 1714658678742437 us; error 78 us; skew 500 ppm
I20240502 14:04:38.743338 29455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:38 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:38.755714 29455 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:04:38.760775 29455 webserver.cc:460] Webserver started at http://127.27.174.126:32813/ using document root <none> and password file <none>
I20240502 14:04:38.761636 29455 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:38.761834 29455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:38.762270 29455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:38.767149 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "5eae78ebf4eb4591986cc850beb0f238"
format_stamp: "Formatted at 2024-05-02 14:04:38 on dist-test-slave-dlb8"
server_key: "4e5b19dde682aed8721d098c35650e3f"
server_key_iv: "596436e44b2289a132938c1d35b6491c"
server_key_version: "encryptionkey@0"
I20240502 14:04:38.768237 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "5eae78ebf4eb4591986cc850beb0f238"
format_stamp: "Formatted at 2024-05-02 14:04:38 on dist-test-slave-dlb8"
server_key: "4e5b19dde682aed8721d098c35650e3f"
server_key_iv: "596436e44b2289a132938c1d35b6491c"
server_key_version: "encryptionkey@0"
I20240502 14:04:38.775174 29455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:04:38.780434 29472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:38.781440 29455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:04:38.781749 29455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "5eae78ebf4eb4591986cc850beb0f238"
format_stamp: "Formatted at 2024-05-02 14:04:38 on dist-test-slave-dlb8"
server_key: "4e5b19dde682aed8721d098c35650e3f"
server_key_iv: "596436e44b2289a132938c1d35b6491c"
server_key_version: "encryptionkey@0"
I20240502 14:04:38.782160 29455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:38.833305 29455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:38.838543 29455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:38.838989 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:38.915993 29455 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:38089
I20240502 14:04:38.916086 29533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:38089 every 8 connection(s)
I20240502 14:04:38.918641 29455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:04:38.923492 29534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:38.926905 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29455
I20240502 14:04:38.927309 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:04:38.928421 28345 external_mini_cluster.cc:1428] Setting key 647133f7cca884f2583723a61f4f2415
I20240502 14:04:38.945803 29534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Bootstrap starting.
May 02 14:04:38 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:38.952100 29534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:38.954380 29534 log.cc:824] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:38.959945 29534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: No bootstrap required, opened a new log
I20240502 14:04:38.963694 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:38.931152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.27.174.126:38089)
0502 14:04:38.934373 (+ 3221us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:38.934400 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:38.934467 (+ 67us) server_negotiation.cc:407] Connection header received
0502 14:04:38.938198 (+ 3731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:38.938244 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:38.938579 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:38.938899 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:38.942352 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:38.943699 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:38.945423 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:38.945804 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:38.951471 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:38.951593 (+ 122us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:38.951635 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:38.951771 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:38.957352 (+ 5581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:38.958662 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:38.958692 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:38.958738 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:38.958959 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:38.959470 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:38.959496 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:38.959517 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:38.960280 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:38.960747 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:38.961258 (+ 511us) server_negotiation.cc:299] Negotiation successful
0502 14:04:38.961872 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2651,"thread_start_us":2179,"threads_started":1}
I20240502 14:04:38.980542 29534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } }
I20240502 14:04:38.981202 29534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:38.981458 29534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eae78ebf4eb4591986cc850beb0f238, State: Initialized, Role: FOLLOWER
I20240502 14:04:38.982179 29534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } }
I20240502 14:04:38.982653 29534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:38.982934 29534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:38.983203 29534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:38.987445 29534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } }
I20240502 14:04:38.988214 29534 leader_election.cc:302] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eae78ebf4eb4591986cc850beb0f238; no voters:
I20240502 14:04:38.989787 29534 leader_election.cc:288] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:38.990161 29539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:38.992579 29539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [term 1 LEADER]: Becoming Leader. State: Replica: 5eae78ebf4eb4591986cc850beb0f238, State: Running, Role: LEADER
I20240502 14:04:38.993320 29539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } }
I20240502 14:04:38.994290 29534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:39.003294 29541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eae78ebf4eb4591986cc850beb0f238. Latest consensus state: current_term: 1 leader_uuid: "5eae78ebf4eb4591986cc850beb0f238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } } }
I20240502 14:04:39.003460 29540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eae78ebf4eb4591986cc850beb0f238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae78ebf4eb4591986cc850beb0f238" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 38089 } } }
I20240502 14:04:39.004151 29541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:39.004216 29540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:39.008047 29547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:39.019279 29547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:39.034471 29547 catalog_manager.cc:1347] Generated new cluster ID: 442a4dab10124868bbe6979baf474264
I20240502 14:04:39.034813 29547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:39.064939 29547 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:39.066995 29547 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:39.089355 29547 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Generated new TSK 0
I20240502 14:04:39.090420 29547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:04:39.146824 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38089
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:39.446015 29560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:39.446547 29560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:39.447209 29560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:39.476867 29560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:39.477247 29560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:39.477888 29560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:39.478459 29560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:04:39.512598 29560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:39.513804 29560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:39.514902 29560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:39.532378 29567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:39.533701 29566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:39.534628 29560 server_base.cc:1034] running on GCE node
W20240502 14:04:39.536679 29569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:40.611161 29560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:40.614449 29560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:40.615795 29560 hybrid_clock.cc:648] HybridClock initialized: now 1714658680615771 us; error 44 us; skew 500 ppm
I20240502 14:04:40.616886 29560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:40 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:40.624255 29560 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:04:40.629837 29560 webserver.cc:460] Webserver started at http://127.27.174.65:44553/ using document root <none> and password file <none>
I20240502 14:04:40.631057 29560 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:40.631327 29560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:40.631870 29560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:40.639334 29560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "265ad9984f2b4ea8928915e248911683"
format_stamp: "Formatted at 2024-05-02 14:04:40 on dist-test-slave-dlb8"
server_key: "d800b67dabee7facd2ca69585aee1aac"
server_key_iv: "0309f5a845c4a0197071a7ce0f9824c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:40.640933 29560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "265ad9984f2b4ea8928915e248911683"
format_stamp: "Formatted at 2024-05-02 14:04:40 on dist-test-slave-dlb8"
server_key: "d800b67dabee7facd2ca69585aee1aac"
server_key_iv: "0309f5a845c4a0197071a7ce0f9824c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:40.650720 29560 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20240502 14:04:40.658404 29577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:40.659581 29560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:04:40.659951 29560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "265ad9984f2b4ea8928915e248911683"
format_stamp: "Formatted at 2024-05-02 14:04:40 on dist-test-slave-dlb8"
server_key: "d800b67dabee7facd2ca69585aee1aac"
server_key_iv: "0309f5a845c4a0197071a7ce0f9824c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:40.660434 29560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:40.717877 29560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:40.723227 29560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:40.723671 29560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:40.733436 29560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:40.733772 29560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:40.734100 29560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:40.734356 29560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:40 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:40.769907 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:40.730030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49143 (local address 127.27.174.126:38089)
0502 14:04:40.731508 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:40.731536 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:40.734265 (+ 2729us) server_negotiation.cc:407] Connection header received
0502 14:04:40.742070 (+ 7805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:40.742105 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:40.742366 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:40.742558 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:40.746001 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.747291 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:40.749531 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.750246 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:40.756085 (+ 5839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:40.756172 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:40.756188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:40.756279 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:40.761798 (+ 5519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:40.763572 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:40.763610 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:40.763637 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:40.763803 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:40.764592 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:40.764616 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:40.764629 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:40.765025 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:40.765301 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:40.766746 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0502 14:04:40.767309 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":542,"threads_started":1}
I20240502 14:04:40.770382 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:40.730164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.65:49143)
0502 14:04:40.733979 (+ 3815us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:40.734085 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:40.741560 (+ 7475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:40.742892 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:40.742933 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:40.744155 (+ 1222us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:40.745596 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:40.745671 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.747567 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:40.747610 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:40.749148 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:40.749208 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.749693 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:40.751019 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:40.751130 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:40.755789 (+ 4659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:40.762114 (+ 6325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:40.762154 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:40.762202 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:40.763340 (+ 1138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:40.763990 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:40.764009 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:40.764023 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:40.764228 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:40.765284 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:40.765315 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:40.765993 (+ 678us) client_negotiation.cc:769] Sending connection context
0502 14:04:40.766502 (+ 509us) client_negotiation.cc:240] Negotiation successful
0502 14:04:40.767224 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3061,"thread_start_us":2214,"threads_started":1}
I20240502 14:04:40.813609 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:40.785754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43085 (local address 127.27.174.126:38089)
0502 14:04:40.786555 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:40.786586 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:40.789402 (+ 2816us) server_negotiation.cc:407] Connection header received
0502 14:04:40.789852 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:40.789884 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:40.790113 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:40.790336 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:40.793051 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.794429 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:40.796697 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.797311 (+ 614us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:40.800079 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:40.800176 (+ 97us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:40.800212 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:40.800354 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:40.806182 (+ 5828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:40.807636 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:40.807671 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:40.807707 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:40.807900 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:40.808786 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:40.808826 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:40.808856 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:40.809412 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:40.809675 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:40.810848 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0502 14:04:40.811347 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20240502 14:04:40.814316 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:40.785642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.65:43085)
0502 14:04:40.789148 (+ 3506us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:40.789226 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:40.789586 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:40.790420 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:40.790457 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:40.791608 (+ 1151us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:40.792711 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:40.792768 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.794680 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:40.794716 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:40.796377 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:40.796432 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:40.796790 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:40.797858 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:40.797956 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:40.799790 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:40.806403 (+ 6613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:40.806430 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:40.806457 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:40.807391 (+ 934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:40.808168 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:40.808216 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:40.808270 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:40.808542 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:40.809687 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:40.809731 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:40.810248 (+ 517us) client_negotiation.cc:769] Sending connection context
0502 14:04:40.810628 (+ 380us) client_negotiation.cc:240] Negotiation successful
0502 14:04:40.811211 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":2181,"threads_started":1}
I20240502 14:04:40.881389 29560 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39075
I20240502 14:04:40.881951 29697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39075 every 8 connection(s)
I20240502 14:04:40.883963 29560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:04:40.888355 29698 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38089
I20240502 14:04:40.888700 29698 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:40.889672 29698 heartbeater.cc:511] Master 127.27.174.126:38089 requested a full tablet report, sending...
I20240502 14:04:40.892086 29489 ts_manager.cc:194] Registered new tserver with Master: 265ad9984f2b4ea8928915e248911683 (127.27.174.65:39075)
I20240502 14:04:40.893962 29489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:49143
I20240502 14:04:40.894220 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29560
I20240502 14:04:40.894655 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:04:40.895941 28345 external_mini_cluster.cc:1428] Setting key f22a9c5781c45586f8e0437270c43086
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:04:40.942603 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38089
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:41.231451 29702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:41.231952 29702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:41.232553 29702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:41.261301 29702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:41.261662 29702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:41.262392 29702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:41.262887 29702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:04:41.296728 29702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:41.297900 29702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:41.298943 29702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:41.316924 29709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:41.897758 29698 heartbeater.cc:503] Master 127.27.174.126:38089 was elected leader, sending a full tablet report...
W20240502 14:04:41.316957 29711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:41.319190 29702 server_base.cc:1034] running on GCE node
W20240502 14:04:41.317907 29708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:42.372114 29702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:42.374846 29702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:42.376259 29702 hybrid_clock.cc:648] HybridClock initialized: now 1714658682376210 us; error 70 us; skew 500 ppm
I20240502 14:04:42.377007 29702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:42 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:42.382452 29702 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:04:42.387981 29702 webserver.cc:460] Webserver started at http://127.27.174.66:38637/ using document root <none> and password file <none>
I20240502 14:04:42.388857 29702 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:42.389061 29702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:42.389529 29702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:42.394482 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c9cf29676e4f4c4fad94cc354f0aaa84"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-dlb8"
server_key: "8d6144bb6b3393612dbbba9056a3cec4"
server_key_iv: "c5afab9abc5393a346276d1a3b8c40d3"
server_key_version: "encryptionkey@0"
I20240502 14:04:42.395675 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c9cf29676e4f4c4fad94cc354f0aaa84"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-dlb8"
server_key: "8d6144bb6b3393612dbbba9056a3cec4"
server_key_iv: "c5afab9abc5393a346276d1a3b8c40d3"
server_key_version: "encryptionkey@0"
I20240502 14:04:42.403048 29702 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20240502 14:04:42.408509 29719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.409616 29702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:04:42.409919 29702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "c9cf29676e4f4c4fad94cc354f0aaa84"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-dlb8"
server_key: "8d6144bb6b3393612dbbba9056a3cec4"
server_key_iv: "c5afab9abc5393a346276d1a3b8c40d3"
server_key_version: "encryptionkey@0"
I20240502 14:04:42.410360 29702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:42.458775 29702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:42.463840 29702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:42.464298 29702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:42.472868 29702 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:42.473151 29702 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.473450 29702 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:42.473645 29702 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:42 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:42.509855 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:42.470654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47755 (local address 127.27.174.126:38089)
0502 14:04:42.472335 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:42.472365 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:42.473993 (+ 1628us) server_negotiation.cc:407] Connection header received
0502 14:04:42.481675 (+ 7682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:42.481695 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:42.481878 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:42.482024 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:42.485512 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.486367 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:42.488547 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.489177 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:42.494615 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:42.494731 (+ 116us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:42.494770 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:42.494972 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:42.501051 (+ 6079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:42.502958 (+ 1907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:42.502996 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:42.503025 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:42.503237 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:42.504100 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:42.504134 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:42.504162 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:42.504769 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:42.505056 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:42.506579 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0502 14:04:42.507165 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":604,"threads_started":1}
I20240502 14:04:42.510111 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:42.470733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.66:47755)
0502 14:04:42.473687 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:42.473769 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:42.481242 (+ 7473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:42.482403 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:42.482451 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:42.483742 (+ 1291us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:42.485158 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:42.485233 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.486593 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:42.486623 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:42.488164 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:42.488241 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.488708 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:42.489908 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:42.490020 (+ 112us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:42.494244 (+ 4224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:42.501431 (+ 7187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:42.501478 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:42.501527 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:42.502693 (+ 1166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:42.503490 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:42.503528 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:42.503562 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:42.503896 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:42.505023 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:42.505052 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:42.505694 (+ 642us) client_negotiation.cc:769] Sending connection context
0502 14:04:42.506292 (+ 598us) client_negotiation.cc:240] Negotiation successful
0502 14:04:42.506999 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2170,"thread_start_us":2002,"threads_started":1}
I20240502 14:04:42.557654 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:42.528193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.66:48533)
0502 14:04:42.532918 (+ 4725us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:42.532987 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:42.533349 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:42.534330 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:42.534377 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:42.535633 (+ 1256us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:42.536939 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:42.537006 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.539141 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:42.539178 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:42.540744 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:42.540801 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.541201 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:42.542295 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:42.542388 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:42.544040 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:42.550650 (+ 6610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:42.550681 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:42.550711 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:42.551585 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:42.552304 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:42.552339 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:42.552374 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:42.552604 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:42.553666 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:42.553699 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:42.554085 (+ 386us) client_negotiation.cc:769] Sending connection context
0502 14:04:42.554425 (+ 340us) client_negotiation.cc:240] Negotiation successful
0502 14:04:42.554947 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3889,"thread_start_us":3245,"threads_started":1}
I20240502 14:04:42.557758 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:42.528340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48533 (local address 127.27.174.126:38089)
0502 14:04:42.529145 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:42.529174 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:42.533171 (+ 3997us) server_negotiation.cc:407] Connection header received
0502 14:04:42.533617 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:42.533638 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:42.533921 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:42.534186 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:42.537316 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.538817 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:42.541108 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:42.541689 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:42.544300 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:42.544409 (+ 109us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:42.544445 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:42.544584 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:42.550390 (+ 5806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:42.551817 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:42.551846 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:42.551878 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:42.552069 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:42.552808 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:42.552836 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:42.552859 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:42.553409 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:42.553692 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:42.554605 (+ 913us) server_negotiation.cc:299] Negotiation successful
0502 14:04:42.555247 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20240502 14:04:42.619457 29702 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:37095
I20240502 14:04:42.620010 29839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:37095 every 8 connection(s)
I20240502 14:04:42.621920 29702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:04:42.626160 29840 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38089
I20240502 14:04:42.626737 29840 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:42.627760 29840 heartbeater.cc:511] Master 127.27.174.126:38089 requested a full tablet report, sending...
I20240502 14:04:42.627916 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29702
I20240502 14:04:42.628345 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:04:42.629371 28345 external_mini_cluster.cc:1428] Setting key a74b6e914119b94b079190ba7c89e4ee
I20240502 14:04:42.630007 29489 ts_manager.cc:194] Registered new tserver with Master: c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095)
I20240502 14:04:42.631474 29489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:47755
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:04:42.674245 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:38089
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:42.964758 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:42.965236 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:42.965950 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:42.995172 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:42.995558 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:42.996165 29844 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:42.996644 29844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:04:43.030490 29844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:43.031551 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:43.032481 29844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:43.047223 29850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:43.634277 29840 heartbeater.cc:503] Master 127.27.174.126:38089 was elected leader, sending a full tablet report...
W20240502 14:04:43.048290 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:43.049422 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:43.049612 29844 server_base.cc:1034] running on GCE node
I20240502 14:04:44.093056 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:44.095228 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:44.096550 29844 hybrid_clock.cc:648] HybridClock initialized: now 1714658684096523 us; error 45 us; skew 500 ppm
I20240502 14:04:44.097303 29844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:44 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:44.107913 29844 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:04:44.112861 29844 webserver.cc:460] Webserver started at http://127.27.174.67:41125/ using document root <none> and password file <none>
I20240502 14:04:44.113775 29844 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:44.113987 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:44.114557 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:44.119632 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "685a30641c824069977b69e5ba55a1c4"
format_stamp: "Formatted at 2024-05-02 14:04:44 on dist-test-slave-dlb8"
server_key: "77b904116f6b521c31f279699250b86b"
server_key_iv: "4b5f7337200eaf52992f3e0a143f66c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:44.120652 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "685a30641c824069977b69e5ba55a1c4"
format_stamp: "Formatted at 2024-05-02 14:04:44 on dist-test-slave-dlb8"
server_key: "77b904116f6b521c31f279699250b86b"
server_key_iv: "4b5f7337200eaf52992f3e0a143f66c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:44.127842 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20240502 14:04:44.132864 29861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:44.133846 29844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:04:44.134151 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "685a30641c824069977b69e5ba55a1c4"
format_stamp: "Formatted at 2024-05-02 14:04:44 on dist-test-slave-dlb8"
server_key: "77b904116f6b521c31f279699250b86b"
server_key_iv: "4b5f7337200eaf52992f3e0a143f66c2"
server_key_version: "encryptionkey@0"
I20240502 14:04:44.134495 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:44.187964 29844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:44.192960 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:44.193421 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:44.203333 29844 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:44.203552 29844 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:44.203806 29844 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:44.204010 29844 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:44 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:44.237470 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.199546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:55911 (local address 127.27.174.126:38089)
0502 14:04:44.201187 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.201225 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.203829 (+ 2604us) server_negotiation.cc:407] Connection header received
0502 14:04:44.209238 (+ 5409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.209269 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.209517 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.209726 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:44.212039 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.213405 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.215815 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.216591 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.222627 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:44.222740 (+ 113us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:44.222772 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:44.222928 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:44.228785 (+ 5857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.230615 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.230644 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.230666 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.230872 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.231810 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.231855 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.231881 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.232461 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:44.232750 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.234267 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.234828 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":567,"threads_started":1}
I20240502 14:04:44.237816 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.199833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.67:55911)
0502 14:04:44.203593 (+ 3760us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.203675 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.208840 (+ 5165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.209925 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.209951 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.210865 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:44.211776 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.211813 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.213711 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.213760 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.215569 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.215618 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.216070 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.217372 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:44.217484 (+ 112us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:44.222242 (+ 4758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:44.229099 (+ 6857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:44.229133 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:44.229176 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:44.230338 (+ 1162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:44.231094 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:44.231141 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:44.231178 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:44.231564 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:44.232742 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:44.232790 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:44.233415 (+ 625us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.233938 (+ 523us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.234674 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3203,"thread_start_us":2342,"threads_started":1}
I20240502 14:04:44.280740 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.255236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.27.174.67:47741)
0502 14:04:44.258196 (+ 2960us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.258249 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.258510 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.259107 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.259133 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.259864 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:44.260580 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.260612 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.261870 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.261890 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.263107 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.263159 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.263512 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.264617 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:44.264704 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:44.266567 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:44.272964 (+ 6397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:44.273005 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:44.273047 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:44.274022 (+ 975us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:44.274830 (+ 808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:44.274866 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:44.274897 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:44.275132 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:44.276302 (+ 1170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:44.276347 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:44.276722 (+ 375us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.277048 (+ 326us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.277689 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":1807,"threads_started":1}
I20240502 14:04:44.281373 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.255321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:47741 (local address 127.27.174.126:38089)
0502 14:04:44.256256 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.256293 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.258375 (+ 2082us) server_negotiation.cc:407] Connection header received
0502 14:04:44.258714 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.258734 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.258888 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.259021 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:44.260816 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.261632 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.263412 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.264053 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.266911 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:44.267046 (+ 135us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:44.267088 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:44.267232 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:44.272735 (+ 5503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.274300 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.274339 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.274371 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.274587 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.275360 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.275388 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.275410 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.276014 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:44.278301 (+ 2287us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.278703 (+ 402us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.279239 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20240502 14:04:44.351151 29844 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:36759
I20240502 14:04:44.351703 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:36759 every 8 connection(s)
I20240502 14:04:44.353474 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:04:44.357709 29982 heartbeater.cc:347] Connected to a master server at 127.27.174.126:38089
I20240502 14:04:44.358019 29982 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:44.359016 29982 heartbeater.cc:511] Master 127.27.174.126:38089 requested a full tablet report, sending...
I20240502 14:04:44.360342 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 29844
I20240502 14:04:44.360793 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:04:44.361174 29488 ts_manager.cc:194] Registered new tserver with Master: 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759)
I20240502 14:04:44.362097 28345 external_mini_cluster.cc:1428] Setting key 5d932e3b454178361bd85343b87a9241
I20240502 14:04:44.363019 29488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:55911
I20240502 14:04:44.367058 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:04:44 dist-test-slave-dlb8 krb5kdc[29441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 02 14:04:44 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:44.432690 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.413149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.27.174.126:38089)
0502 14:04:44.413841 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.413878 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.415143 (+ 1265us) server_negotiation.cc:407] Connection header received
0502 14:04:44.415498 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.415519 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.415675 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.415825 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:44.418319 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.419144 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.420476 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.421008 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.424955 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:44.425031 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:44.425069 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:44.425166 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:44.428395 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.429417 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.429436 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.429450 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.429579 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:44.429980 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:44.429996 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:44.430010 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:44.430420 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:44.430679 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.431163 (+ 484us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.431493 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20240502 14:04:44.438616 29488 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:44.458366 29488 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:44.499773 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.484012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39075 (local address 127.0.0.1:55702)
0502 14:04:44.489792 (+ 5780us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.489850 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.490235 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.491872 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.491932 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.492089 (+ 157us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:44.493086 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.493131 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.495449 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.495476 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.497393 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.497427 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.497701 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.497828 (+ 127us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.498000 (+ 172us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.498253 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5105,"thread_start_us":2428,"threads_started":1}
I20240502 14:04:44.500838 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.485618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.27.174.65:39075)
0502 14:04:44.488465 (+ 2847us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.488505 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.489978 (+ 1473us) server_negotiation.cc:407] Connection header received
0502 14:04:44.490624 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.490666 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.491089 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.491321 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:44.493383 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.495135 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.497636 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.498912 (+ 1276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.499143 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.499341 (+ 198us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.499565 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":2824,"threads_started":1}
I20240502 14:04:44.502302 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.482916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36759 (local address 127.0.0.1:59706)
0502 14:04:44.486242 (+ 3326us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.486331 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.486653 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.494353 (+ 7700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.494380 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.494469 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:44.495292 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.495340 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.498210 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.498244 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.499936 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.499980 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.500246 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.500323 (+ 77us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.500433 (+ 110us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.500632 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2642,"thread_start_us":10901,"threads_started":1}
I20240502 14:04:44.505247 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.483374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.27.174.67:36759)
0502 14:04:44.493441 (+ 10067us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.493472 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.493525 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:04:44.493702 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.493732 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.494068 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.495597 (+ 1529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:44.496102 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.497901 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.501275 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.502762 (+ 1487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.502980 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.503159 (+ 179us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.503356 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9455,"thread_start_us":9044,"threads_started":1}
I20240502 14:04:44.504894 29633 tablet_service.cc:1467] Processing CreateTablet for tablet 227d1d24e89d443bb60846978fdd880c (DEFAULT_TABLE table=test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:44.506763 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227d1d24e89d443bb60846978fdd880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:44.507146 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.487423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37095 (local address 127.0.0.1:39228)
0502 14:04:44.494306 (+ 6883us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.494398 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.494692 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.496031 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.496061 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.496153 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:44.497104 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.497168 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.502170 (+ 5002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.502199 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.503864 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.503915 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.504199 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.504286 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.504404 (+ 118us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.504592 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6317,"spinlock_wait_cycles":12672,"thread_start_us":2421,"threads_started":1}
I20240502 14:04:44.509179 29917 tablet_service.cc:1467] Processing CreateTablet for tablet 227d1d24e89d443bb60846978fdd880c (DEFAULT_TABLE table=test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:44.509631 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.489025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.27.174.66:37095)
0502 14:04:44.493828 (+ 4803us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.493856 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.494873 (+ 1017us) server_negotiation.cc:407] Connection header received
0502 14:04:44.495035 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.495073 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.495510 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.495692 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:44.497427 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.501578 (+ 4151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.505766 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.507142 (+ 1376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.507493 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.507978 (+ 485us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.508211 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4230,"thread_start_us":3937,"threads_started":1}
I20240502 14:04:44.510759 29917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227d1d24e89d443bb60846978fdd880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:44.513458 29775 tablet_service.cc:1467] Processing CreateTablet for tablet 227d1d24e89d443bb60846978fdd880c (DEFAULT_TABLE table=test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:44.515420 29775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227d1d24e89d443bb60846978fdd880c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:44.535318 30001 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Bootstrap starting.
I20240502 14:04:44.538563 30002 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Bootstrap starting.
I20240502 14:04:44.542852 30001 tablet_bootstrap.cc:654] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:44.544114 30003 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Bootstrap starting.
I20240502 14:04:44.545368 30002 tablet_bootstrap.cc:654] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:44.545387 30001 log.cc:824] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:44.547214 30002 log.cc:824] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:44.551905 30001 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: No bootstrap required, opened a new log
I20240502 14:04:44.552323 30001 ts_tablet_manager.cc:1367] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.002s
I20240502 14:04:44.552621 30003 tablet_bootstrap.cc:654] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:44.552793 30002 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: No bootstrap required, opened a new log
I20240502 14:04:44.553260 30002 ts_tablet_manager.cc:1367] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240502 14:04:44.554881 30003 log.cc:824] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:44.560925 30003 tablet_bootstrap.cc:492] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: No bootstrap required, opened a new log
I20240502 14:04:44.561450 30003 ts_tablet_manager.cc:1367] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240502 14:04:44.574375 30001 raft_consensus.cc:348] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.575017 30001 raft_consensus.cc:374] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:44.575271 30001 raft_consensus.cc:729] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685a30641c824069977b69e5ba55a1c4, State: Initialized, Role: FOLLOWER
I20240502 14:04:44.575956 30001 consensus_queue.cc:260] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.579041 29982 heartbeater.cc:503] Master 127.27.174.126:38089 was elected leader, sending a full tablet report...
I20240502 14:04:44.579550 30001 ts_tablet_manager.cc:1398] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Time spent starting tablet: real 0.027s user 0.017s sys 0.011s
I20240502 14:04:44.581861 30002 raft_consensus.cc:348] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.582777 30002 raft_consensus.cc:374] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:44.583092 30002 raft_consensus.cc:729] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265ad9984f2b4ea8928915e248911683, State: Initialized, Role: FOLLOWER
I20240502 14:04:44.584007 30002 consensus_queue.cc:260] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.589481 30002 ts_tablet_manager.cc:1398] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Time spent starting tablet: real 0.036s user 0.036s sys 0.000s
I20240502 14:04:44.590214 30003 raft_consensus.cc:348] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.591066 30003 raft_consensus.cc:374] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:44.591368 30003 raft_consensus.cc:729] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cf29676e4f4c4fad94cc354f0aaa84, State: Initialized, Role: FOLLOWER
I20240502 14:04:44.592120 30003 consensus_queue.cc:260] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.595706 30003 ts_tablet_manager.cc:1398] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent starting tablet: real 0.034s user 0.028s sys 0.006s
W20240502 14:04:44.607124 29983 tablet.cc:2369] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:04:44.628607 29841 tablet.cc:2369] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:04:44.642163 29699 tablet.cc:2369] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:44.842547 30008 raft_consensus.cc:482] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:44.842993 30008 raft_consensus.cc:504] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.845255 30008 leader_election.cc:288] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759), c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095)
I20240502 14:04:44.857620 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.845779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37095 (local address 127.27.174.65:49923)
0502 14:04:44.848707 (+ 2928us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.848751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.849047 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.849953 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.849987 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.850117 (+ 130us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:44.851198 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.851246 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.852841 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.852868 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.854725 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.854773 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.855095 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.855182 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.855308 (+ 126us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.855501 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2300,"spinlock_wait_cycles":27904,"thread_start_us":1977,"threads_started":1}
I20240502 14:04:44.857638 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.845779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36759 (local address 127.27.174.65:52809)
0502 14:04:44.848669 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:44.848701 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:44.848990 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:44.849769 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:44.849811 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:44.849916 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:44.850972 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.851046 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.852807 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.852840 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:44.854727 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:44.854773 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.855095 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.855182 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:04:44.855308 (+ 126us) client_negotiation.cc:240] Negotiation successful
0502 14:04:44.855500 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2437,"mutex_wait_us":58,"thread_start_us":2212,"threads_started":1}
I20240502 14:04:44.858036 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.846230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49923 (local address 127.27.174.66:37095)
0502 14:04:44.847192 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.847227 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.848892 (+ 1665us) server_negotiation.cc:407] Connection header received
0502 14:04:44.849389 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.849418 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.849674 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.849828 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:44.851458 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.852602 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.855009 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.856138 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.856282 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.856456 (+ 174us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.856658 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20240502 14:04:44.858578 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.845944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:52809 (local address 127.27.174.67:36759)
0502 14:04:44.846753 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.846783 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.848846 (+ 2063us) server_negotiation.cc:407] Connection header received
0502 14:04:44.849251 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.849284 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.849540 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.849703 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:44.851259 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.852461 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.855002 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.856137 (+ 1135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.856281 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.856465 (+ 184us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.856658 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20240502 14:04:44.859788 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227d1d24e89d443bb60846978fdd880c" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" is_pre_election: true
I20240502 14:04:44.859979 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227d1d24e89d443bb60846978fdd880c" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4" is_pre_election: true
I20240502 14:04:44.860724 29937 raft_consensus.cc:2454] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 0.
I20240502 14:04:44.860724 29795 raft_consensus.cc:2454] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 0.
I20240502 14:04:44.861922 29581 leader_election.cc:302] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265ad9984f2b4ea8928915e248911683, 685a30641c824069977b69e5ba55a1c4; no voters:
I20240502 14:04:44.862624 30008 raft_consensus.cc:2781] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:44.862882 30008 raft_consensus.cc:482] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:44.863114 30008 raft_consensus.cc:3037] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.867743 30008 raft_consensus.cc:504] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.869084 30008 leader_election.cc:288] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 election: Requested vote from peers 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759), c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095)
I20240502 14:04:44.869935 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227d1d24e89d443bb60846978fdd880c" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4"
I20240502 14:04:44.870103 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227d1d24e89d443bb60846978fdd880c" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84"
I20240502 14:04:44.870379 29937 raft_consensus.cc:3037] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.870554 29795 raft_consensus.cc:3037] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.875514 29795 raft_consensus.cc:2454] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 1.
I20240502 14:04:44.875627 29937 raft_consensus.cc:2454] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 1.
I20240502 14:04:44.876467 29579 leader_election.cc:302] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265ad9984f2b4ea8928915e248911683, c9cf29676e4f4c4fad94cc354f0aaa84; no voters:
I20240502 14:04:44.877126 30008 raft_consensus.cc:2781] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:44.878633 30008 raft_consensus.cc:686] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 1 LEADER]: Becoming Leader. State: Replica: 265ad9984f2b4ea8928915e248911683, State: Running, Role: LEADER
I20240502 14:04:44.879333 30008 consensus_queue.cc:237] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } }
I20240502 14:04:44.890772 29489 catalog_manager.cc:5484] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265ad9984f2b4ea8928915e248911683 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "265ad9984f2b4ea8928915e248911683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:44.985873 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:44.977924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.27.174.65:39075)
0502 14:04:44.978410 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:44.978428 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:44.980198 (+ 1770us) server_negotiation.cc:407] Connection header received
0502 14:04:44.980488 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:44.980506 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:44.980712 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:44.980841 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:44.981601 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.982387 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:44.983674 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:44.983993 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:44.984120 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:44.984628 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:44.984751 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:44.984934 (+ 183us) server_negotiation.cc:299] Negotiation successful
0502 14:04:44.985040 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20240502 14:04:44.997464 29795 raft_consensus.cc:1261] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Refusing update from remote peer 265ad9984f2b4ea8928915e248911683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:44.997838 29937 raft_consensus.cc:1261] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Refusing update from remote peer 265ad9984f2b4ea8928915e248911683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:44.999126 30008 consensus_queue.cc:1035] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:45.000166 30012 consensus_queue.cc:1035] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:45.019534 30017 mvcc.cc:205] Tried to move back new op lower bound from 7023241973735235584 to 7023241973272965120. Current Snapshot: MvccSnapshot[applied={T|T < 7023241973735235584}]
I20240502 14:04:45.019987 30016 mvcc.cc:205] Tried to move back new op lower bound from 7023241973735235584 to 7023241973272965120. Current Snapshot: MvccSnapshot[applied={T|T < 7023241973735235584}]
I20240502 14:04:45.050873 29488 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:45.051343 29488 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:45.059453 29488 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Sending DeleteTablet for 3 replicas of tablet 227d1d24e89d443bb60846978fdd880c
I20240502 14:04:45.061317 29917 tablet_service.cc:1514] Processing DeleteTablet for tablet 227d1d24e89d443bb60846978fdd880c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:45 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59706
I20240502 14:04:45.061636 29633 tablet_service.cc:1514] Processing DeleteTablet for tablet 227d1d24e89d443bb60846978fdd880c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:45 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:55702
I20240502 14:04:45.062222 29775 tablet_service.cc:1514] Processing DeleteTablet for tablet 227d1d24e89d443bb60846978fdd880c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:45 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:39228
I20240502 14:04:45.064005 29488 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:45.066771 29488 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:45.077355 30026 tablet_replica.cc:330] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: stopping tablet replica
I20240502 14:04:45.078606 30026 raft_consensus.cc:2229] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:45.079596 30026 raft_consensus.cc:2258] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:45.080986 30028 tablet_replica.cc:330] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: stopping tablet replica
I20240502 14:04:45.081004 30027 tablet_replica.cc:330] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: stopping tablet replica
I20240502 14:04:45.081804 30027 raft_consensus.cc:2229] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:45.081895 30028 raft_consensus.cc:2229] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:45.082712 30028 raft_consensus.cc:2258] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:45.082829 30027 raft_consensus.cc:2258] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:45.086580 30028 ts_tablet_manager.cc:1875] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:45.087611 30027 ts_tablet_manager.cc:1875] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:45.089608 30026 ts_tablet_manager.cc:1875] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:45.093835 29633 tablet_service.cc:1467] Processing CreateTablet for tablet 01137234376e47fd8adf03fb9785005a (DEFAULT_TABLE table=test-table [id=3e58edfc5f134864823657b3d845516a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:45.094954 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01137234376e47fd8adf03fb9785005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.096320 29775 tablet_service.cc:1467] Processing CreateTablet for tablet 01137234376e47fd8adf03fb9785005a (DEFAULT_TABLE table=test-table [id=3e58edfc5f134864823657b3d845516a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:45.097424 29775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01137234376e47fd8adf03fb9785005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.104566 29917 tablet_service.cc:1467] Processing CreateTablet for tablet 01137234376e47fd8adf03fb9785005a (DEFAULT_TABLE table=test-table [id=3e58edfc5f134864823657b3d845516a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:45.105705 29917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01137234376e47fd8adf03fb9785005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.108050 30026 ts_tablet_manager.cc:1888] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:04:45.114240 30026 log.cc:1196] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/227d1d24e89d443bb60846978fdd880c
I20240502 14:04:45.120805 30026 ts_tablet_manager.cc:1909] T 227d1d24e89d443bb60846978fdd880c P 685a30641c824069977b69e5ba55a1c4: Deleting consensus metadata
I20240502 14:04:45.123920 29476 catalog_manager.cc:4830] TS 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759): tablet 227d1d24e89d443bb60846978fdd880c (table test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]) successfully deleted
I20240502 14:04:45.125262 30029 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Bootstrap starting.
I20240502 14:04:45.124110 30028 ts_tablet_manager.cc:1888] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:04:45.125838 30028 log.cc:1196] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/227d1d24e89d443bb60846978fdd880c
I20240502 14:04:45.126750 30028 ts_tablet_manager.cc:1909] T 227d1d24e89d443bb60846978fdd880c P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting consensus metadata
I20240502 14:04:45.133961 29474 catalog_manager.cc:4830] TS c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095): tablet 227d1d24e89d443bb60846978fdd880c (table test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]) successfully deleted
I20240502 14:04:45.139724 30030 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Bootstrap starting.
I20240502 14:04:45.139801 30029 tablet_bootstrap.cc:654] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.143944 30031 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Bootstrap starting.
I20240502 14:04:45.146800 30030 tablet_bootstrap.cc:654] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.149376 30027 ts_tablet_manager.cc:1888] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:04:45.149844 30027 log.cc:1196] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/227d1d24e89d443bb60846978fdd880c
I20240502 14:04:45.150722 30027 ts_tablet_manager.cc:1909] T 227d1d24e89d443bb60846978fdd880c P 265ad9984f2b4ea8928915e248911683: Deleting consensus metadata
I20240502 14:04:45.151643 30031 tablet_bootstrap.cc:654] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.153673 29474 catalog_manager.cc:4830] TS 265ad9984f2b4ea8928915e248911683 (127.27.174.65:39075): tablet 227d1d24e89d443bb60846978fdd880c (table test-table [id=8a1c314d754b40ca86e6c9a2e7f30ade]) successfully deleted
I20240502 14:04:45.155830 30029 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: No bootstrap required, opened a new log
I20240502 14:04:45.156231 30029 ts_tablet_manager.cc:1367] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20240502 14:04:45.158804 30029 raft_consensus.cc:348] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.159482 30029 raft_consensus.cc:374] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.159734 30029 raft_consensus.cc:729] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685a30641c824069977b69e5ba55a1c4, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.160336 30029 consensus_queue.cc:260] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.164991 30029 ts_tablet_manager.cc:1398] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:04:45.165277 30031 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: No bootstrap required, opened a new log
I20240502 14:04:45.165791 30031 ts_tablet_manager.cc:1367] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20240502 14:04:45.168506 30030 tablet_bootstrap.cc:492] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: No bootstrap required, opened a new log
I20240502 14:04:45.168391 30031 raft_consensus.cc:348] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.168982 30030 ts_tablet_manager.cc:1367] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.009s
I20240502 14:04:45.169122 30031 raft_consensus.cc:374] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.169415 30031 raft_consensus.cc:729] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cf29676e4f4c4fad94cc354f0aaa84, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.170050 30031 consensus_queue.cc:260] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.171664 30030 raft_consensus.cc:348] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.172389 30030 raft_consensus.cc:374] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.172677 30030 raft_consensus.cc:729] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265ad9984f2b4ea8928915e248911683, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.173462 30031 ts_tablet_manager.cc:1398] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:04:45.173281 30030 consensus_queue.cc:260] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.175621 30030 ts_tablet_manager.cc:1398] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:04:45.200855 30023 raft_consensus.cc:482] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:45.201313 30023 raft_consensus.cc:504] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.203449 30023 leader_election.cc:288] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095), 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759)
I20240502 14:04:45.204242 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01137234376e47fd8adf03fb9785005a" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" is_pre_election: true
I20240502 14:04:45.204412 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01137234376e47fd8adf03fb9785005a" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4" is_pre_election: true
I20240502 14:04:45.204912 29937 raft_consensus.cc:2454] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 0.
I20240502 14:04:45.204869 29795 raft_consensus.cc:2454] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 0.
I20240502 14:04:45.205695 29581 leader_election.cc:302] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265ad9984f2b4ea8928915e248911683, 685a30641c824069977b69e5ba55a1c4; no voters:
I20240502 14:04:45.206665 30023 raft_consensus.cc:2781] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:45.207020 30023 raft_consensus.cc:482] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:45.207324 30023 raft_consensus.cc:3037] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.213687 30023 raft_consensus.cc:504] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.216230 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01137234376e47fd8adf03fb9785005a" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84"
I20240502 14:04:45.216411 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01137234376e47fd8adf03fb9785005a" candidate_uuid: "265ad9984f2b4ea8928915e248911683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4"
I20240502 14:04:45.216822 29937 raft_consensus.cc:3037] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.216773 29795 raft_consensus.cc:3037] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.218336 30023 leader_election.cc:288] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 election: Requested vote from peers c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095), 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759)
I20240502 14:04:45.221029 29937 raft_consensus.cc:2454] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 1.
I20240502 14:04:45.221853 29581 leader_election.cc:302] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265ad9984f2b4ea8928915e248911683, 685a30641c824069977b69e5ba55a1c4; no voters:
I20240502 14:04:45.222469 30023 raft_consensus.cc:2781] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:45.222807 30023 raft_consensus.cc:686] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 1 LEADER]: Becoming Leader. State: Replica: 265ad9984f2b4ea8928915e248911683, State: Running, Role: LEADER
I20240502 14:04:45.223134 29795 raft_consensus.cc:2454] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265ad9984f2b4ea8928915e248911683 in term 1.
I20240502 14:04:45.223594 30023 consensus_queue.cc:237] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.231528 29488 catalog_manager.cc:5484] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 265ad9984f2b4ea8928915e248911683 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "265ad9984f2b4ea8928915e248911683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } health_report { overall_health: UNKNOWN } } }
May 02 14:04:45 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:45.311952 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.289332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:38089 (local address 127.0.0.1:51596)
0502 14:04:45.291900 (+ 2568us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.291960 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.292254 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.292972 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.292994 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.293792 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:45.294233 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.294265 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.295594 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.295615 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.296466 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.296502 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.296753 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.297704 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:45.297772 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:45.300776 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:45.306775 (+ 5999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:45.306804 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:45.306820 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:45.307635 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:45.308320 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:45.308341 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:45.308355 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:45.308525 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:45.309481 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:45.309505 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:45.309741 (+ 236us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.309949 (+ 208us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.310446 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":1624,"threads_started":1}
I20240502 14:04:45.313040 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.289531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.27.174.126:38089)
0502 14:04:45.292471 (+ 2940us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.292485 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.292514 (+ 29us) server_negotiation.cc:407] Connection header received
0502 14:04:45.292597 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.292611 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.292747 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.292879 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:45.294473 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.295391 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.296672 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.297210 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.301040 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:45.301140 (+ 100us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:45.301167 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:45.301291 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:45.306551 (+ 5260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:45.307872 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:45.307905 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:45.307930 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:45.308144 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:45.308714 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:45.308746 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:45.308768 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:45.309280 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:45.309539 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.310243 (+ 704us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.310774 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":2370,"thread_start_us":2116,"threads_started":1}
I20240502 14:04:45.320453 29488 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:51596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240502 14:04:45.322356 29488 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:45.339884 29633 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b48f9326142e3949a7ee497a3e5e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c8c452d55e449fa94e002119fd6d3c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:45.341181 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b48f9326142e3949a7ee497a3e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.340947 29775 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b48f9326142e3949a7ee497a3e5e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c8c452d55e449fa94e002119fd6d3c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:45.342216 29775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b48f9326142e3949a7ee497a3e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.343014 29917 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b48f9326142e3949a7ee497a3e5e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c8c452d55e449fa94e002119fd6d3c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:45.344111 29917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b48f9326142e3949a7ee497a3e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:45.357681 30030 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: Bootstrap starting.
I20240502 14:04:45.363201 30030 tablet_bootstrap.cc:654] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.364105 30029 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: Bootstrap starting.
I20240502 14:04:45.369599 30029 tablet_bootstrap.cc:654] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.391345 30031 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: Bootstrap starting.
I20240502 14:04:45.392593 30030 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: No bootstrap required, opened a new log
I20240502 14:04:45.393131 30030 ts_tablet_manager.cc:1367] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.012s
I20240502 14:04:45.397315 30031 tablet_bootstrap.cc:654] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:45.397213 30030 raft_consensus.cc:348] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.398070 30030 raft_consensus.cc:374] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.398420 30030 raft_consensus.cc:729] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265ad9984f2b4ea8928915e248911683, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.399180 30030 consensus_queue.cc:260] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.400539 30023 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.401276 30023 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.402539 30029 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: No bootstrap required, opened a new log
I20240502 14:04:45.402987 30029 ts_tablet_manager.cc:1367] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.006s
I20240502 14:04:45.405483 30029 raft_consensus.cc:348] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.406273 30029 raft_consensus.cc:374] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.406603 30029 raft_consensus.cc:729] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685a30641c824069977b69e5ba55a1c4, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.407367 30029 consensus_queue.cc:260] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.408730 30007 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.409561 30007 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.409893 30030 ts_tablet_manager.cc:1398] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: Time spent starting tablet: real 0.016s user 0.007s sys 0.002s
I20240502 14:04:45.415774 30029 ts_tablet_manager.cc:1398] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: Time spent starting tablet: real 0.012s user 0.003s sys 0.001s
I20240502 14:04:45.417655 30031 tablet_bootstrap.cc:492] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: No bootstrap required, opened a new log
I20240502 14:04:45.418243 30031 ts_tablet_manager.cc:1367] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.000s
I20240502 14:04:45.420717 30031 raft_consensus.cc:348] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.421499 30031 raft_consensus.cc:374] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:45.421835 30031 raft_consensus.cc:729] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cf29676e4f4c4fad94cc354f0aaa84, State: Initialized, Role: FOLLOWER
I20240502 14:04:45.422734 30031 consensus_queue.cc:260] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.424057 30009 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.424845 30009 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.426720 30031 ts_tablet_manager.cc:1398] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:04:45.477921 30009 raft_consensus.cc:482] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:45.478437 30009 raft_consensus.cc:504] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.484062 30009 leader_election.cc:288] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 265ad9984f2b4ea8928915e248911683 (127.27.174.65:39075), 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759)
I20240502 14:04:45.505501 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.481328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39075 (local address 127.27.174.66:53675)
0502 14:04:45.496513 (+ 15185us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.496564 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.497259 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.498440 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.498475 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.498586 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:45.499533 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.499586 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.501451 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.501506 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.503169 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.503219 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.503497 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.503574 (+ 77us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.503701 (+ 127us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.503882 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14565,"thread_start_us":18061,"threads_started":1}
I20240502 14:04:45.506332 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.482421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53675 (local address 127.27.174.65:39075)
0502 14:04:45.483201 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.483226 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.496689 (+ 13463us) server_negotiation.cc:407] Connection header received
0502 14:04:45.497589 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.497620 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.497873 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.498005 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:45.499830 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.501119 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.503405 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.504478 (+ 1073us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.504617 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.504788 (+ 171us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.504973 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20240502 14:04:45.507921 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b48f9326142e3949a7ee497a3e5e8" candidate_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265ad9984f2b4ea8928915e248911683" is_pre_election: true
I20240502 14:04:45.508467 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.481619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36759 (local address 127.27.174.66:53135)
0502 14:04:45.498983 (+ 17364us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.499030 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.499305 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.500049 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.500135 (+ 86us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.500235 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:45.501063 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.501097 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.503273 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.503301 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.504982 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.505026 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.505311 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.505403 (+ 92us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.506515 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.506762 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16914,"thread_start_us":15018,"threads_started":1}
I20240502 14:04:45.508754 29653 raft_consensus.cc:2454] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cf29676e4f4c4fad94cc354f0aaa84 in term 0.
I20240502 14:04:45.509989 29721 leader_election.cc:302] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265ad9984f2b4ea8928915e248911683, c9cf29676e4f4c4fad94cc354f0aaa84; no voters:
I20240502 14:04:45.510309 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.481815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53135 (local address 127.27.174.67:36759)
0502 14:04:45.498010 (+ 16195us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.498043 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.499193 (+ 1150us) server_negotiation.cc:407] Connection header received
0502 14:04:45.499543 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.499570 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.499815 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.499963 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:45.501329 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.503013 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.505677 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.507188 (+ 1511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.507330 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.507524 (+ 194us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.507731 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15278,"thread_start_us":16169,"threads_started":1}
I20240502 14:04:45.510892 30009 raft_consensus.cc:2781] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:45.511210 30009 raft_consensus.cc:482] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:45.511521 30009 raft_consensus.cc:3037] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.511529 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b48f9326142e3949a7ee497a3e5e8" candidate_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4" is_pre_election: true
I20240502 14:04:45.512025 29937 raft_consensus.cc:2454] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9cf29676e4f4c4fad94cc354f0aaa84 in term 0.
I20240502 14:04:45.518455 30009 raft_consensus.cc:504] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.520619 30009 leader_election.cc:288] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [CANDIDATE]: Term 1 election: Requested vote from peers 265ad9984f2b4ea8928915e248911683 (127.27.174.65:39075), 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759)
I20240502 14:04:45.521093 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b48f9326142e3949a7ee497a3e5e8" candidate_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265ad9984f2b4ea8928915e248911683"
I20240502 14:04:45.521188 29937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b48f9326142e3949a7ee497a3e5e8" candidate_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685a30641c824069977b69e5ba55a1c4"
I20240502 14:04:45.521667 29653 raft_consensus.cc:3037] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.521679 29937 raft_consensus.cc:3037] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:45.528203 29937 raft_consensus.cc:2454] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cf29676e4f4c4fad94cc354f0aaa84 in term 1.
I20240502 14:04:45.529054 29653 raft_consensus.cc:2454] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9cf29676e4f4c4fad94cc354f0aaa84 in term 1.
I20240502 14:04:45.529340 29723 leader_election.cc:302] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685a30641c824069977b69e5ba55a1c4, c9cf29676e4f4c4fad94cc354f0aaa84; no voters:
I20240502 14:04:45.530126 30009 raft_consensus.cc:2781] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:45.531442 30009 raft_consensus.cc:686] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 LEADER]: Becoming Leader. State: Replica: c9cf29676e4f4c4fad94cc354f0aaa84, State: Running, Role: LEADER
I20240502 14:04:45.532260 30009 consensus_queue.cc:237] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } }
I20240502 14:04:45.538691 30049 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: TxnStatusTablet state changed. Reason: New leader c9cf29676e4f4c4fad94cc354f0aaa84. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.539399 30049 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:45.541425 29487 catalog_manager.cc:5484] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:45.543042 30051 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240502 14:04:45.543438 30051 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240502 14:04:45.585242 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.575289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37095 (local address 127.0.0.1:39244)
0502 14:04:45.576036 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.576083 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.576388 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.578546 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.578584 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.578640 (+ 56us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:45.579083 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.579117 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.580774 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.580800 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.581589 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.581625 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.581922 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.582077 (+ 155us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.583298 (+ 1221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.583345 (+ 47us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.583470 (+ 125us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.583696 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20240502 14:04:45.585445 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.575461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.27.174.66:37095)
0502 14:04:45.577781 (+ 2320us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.577800 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.577835 (+ 35us) server_negotiation.cc:407] Connection header received
0502 14:04:45.577993 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.578009 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.578297 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.578471 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:45.579345 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.580564 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.581841 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.582386 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.582529 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.583123 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.583290 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.583541 (+ 251us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.583696 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1608,"threads_started":1}
I20240502 14:04:45.594353 29937 raft_consensus.cc:1261] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Refusing update from remote peer c9cf29676e4f4c4fad94cc354f0aaa84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:45.594765 29653 raft_consensus.cc:1261] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Refusing update from remote peer c9cf29676e4f4c4fad94cc354f0aaa84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:45.596160 30009 consensus_queue.cc:1035] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:45.597313 30049 consensus_queue.cc:1035] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:45.616228 30007 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: TxnStatusTablet state changed. Reason: New leader c9cf29676e4f4c4fad94cc354f0aaa84. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.616925 30007 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.634264 30055 mvcc.cc:205] Tried to move back new op lower bound from 7023241976180629504 to 7023241975947751424. Current Snapshot: MvccSnapshot[applied={T|T < 7023241976180629504}]
I20240502 14:04:45.636909 30023 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: TxnStatusTablet state changed. Reason: New leader c9cf29676e4f4c4fad94cc354f0aaa84. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.637646 30023 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.648414 30007 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.649150 30007 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 685a30641c824069977b69e5ba55a1c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.649109 30012 consensus_queue.cc:1035] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:04:45.656472 30009 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.657285 30009 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:45.662339 30012 consensus_queue.cc:1035] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:45.666430 30023 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.674597 30023 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P 265ad9984f2b4ea8928915e248911683: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:45.699379 30009 tablet_replica.cc:439] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265ad9984f2b4ea8928915e248911683" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39075 } } peers { permanent_uuid: "c9cf29676e4f4c4fad94cc354f0aaa84" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 37095 } } peers { permanent_uuid: "685a30641c824069977b69e5ba55a1c4" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36759 } } }
I20240502 14:04:45.700119 30009 tablet_replica.cc:442] T 1f9b48f9326142e3949a7ee497a3e5e8 P c9cf29676e4f4c4fad94cc354f0aaa84: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:45.784677 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.763072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:45329 (local address 127.27.174.66:37095)
0502 14:04:45.763779 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.763804 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.774802 (+ 10998us) server_negotiation.cc:407] Connection header received
0502 14:04:45.775711 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.775738 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.776016 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.776149 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:45.777821 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.779103 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.781072 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.781597 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.782349 (+ 752us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.782643 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.782772 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.783183 (+ 411us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.783331 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20240502 14:04:45.787570 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.763508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:37095 (local address 127.27.174.65:45329)
0502 14:04:45.774602 (+ 11094us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.774662 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.775432 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.776585 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.776612 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.776664 (+ 52us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:45.777515 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.777564 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.779435 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.779468 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.780796 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.780843 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.781891 (+ 1048us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.782025 (+ 134us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.782923 (+ 898us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.782969 (+ 46us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.785612 (+ 2643us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.785816 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10372,"thread_start_us":11280,"threads_started":1}
I20240502 14:04:45.904006 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.877180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47989 (local address 127.27.174.65:39075)
0502 14:04:45.877885 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:45.877912 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:45.892898 (+ 14986us) server_negotiation.cc:407] Connection header received
0502 14:04:45.893464 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:45.893490 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:45.893763 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:45.893882 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:45.896906 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.898186 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.900258 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.900784 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.901493 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.901801 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.901918 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:45.902334 (+ 416us) server_negotiation.cc:299] Negotiation successful
0502 14:04:45.902506 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20240502 14:04:45.906908 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:45.878715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39075 (local address 127.27.174.66:47989)
0502 14:04:45.892724 (+ 14009us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:45.892783 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:45.893178 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:45.894325 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:45.894361 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:45.894426 (+ 65us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:45.895490 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.895539 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.898548 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:45.898584 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:45.899973 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:45.900026 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:45.901059 (+ 1033us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:45.901182 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:45.902085 (+ 903us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:45.902120 (+ 35us) client_negotiation.cc:769] Sending connection context
0502 14:04:45.904839 (+ 2719us) client_negotiation.cc:240] Negotiation successful
0502 14:04:45.905070 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13308,"thread_start_us":12897,"threads_started":1}
I20240502 14:04:46.089500 29489 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:46.090113 29489 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:46.120111 29489 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5eae78ebf4eb4591986cc850beb0f238: Sending DeleteTablet for 3 replicas of tablet 01137234376e47fd8adf03fb9785005a
I20240502 14:04:46.123909 29917 tablet_service.cc:1514] Processing DeleteTablet for tablet 01137234376e47fd8adf03fb9785005a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:46 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59706
I20240502 14:04:46.123952 29633 tablet_service.cc:1514] Processing DeleteTablet for tablet 01137234376e47fd8adf03fb9785005a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:46 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:55702
I20240502 14:04:46.124810 29775 tablet_service.cc:1514] Processing DeleteTablet for tablet 01137234376e47fd8adf03fb9785005a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:46 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:39228
I20240502 14:04:46.143141 30078 tablet_replica.cc:330] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: stopping tablet replica
I20240502 14:04:46.143951 30078 raft_consensus.cc:2229] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:46.144429 30078 raft_consensus.cc:2258] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:46.145673 30077 tablet_replica.cc:330] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: stopping tablet replica
I20240502 14:04:46.146577 30077 raft_consensus.cc:2229] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:46.147929 30076 tablet_replica.cc:330] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: stopping tablet replica
I20240502 14:04:46.150578 30077 raft_consensus.cc:2258] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:46.153044 30078 ts_tablet_manager.cc:1875] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:46.153844 30076 raft_consensus.cc:2229] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:46.156373 30076 raft_consensus.cc:2258] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:46.159222 30076 ts_tablet_manager.cc:1875] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:46.166359 30077 ts_tablet_manager.cc:1875] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Deleting tablet data with delete state TABLET_DATA_DELETED
May 02 14:04:46 dist-test-slave-dlb8 krb5kdc[29441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:04:46.182265 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:46.144256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.27.174.65:39075)
0502 14:04:46.144941 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:46.144967 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:46.153379 (+ 8412us) server_negotiation.cc:407] Connection header received
0502 14:04:46.154287 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:46.154316 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:46.154550 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:46.154704 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:46.158775 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:46.159868 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:46.161723 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:46.162395 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:46.169349 (+ 6954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:46.169465 (+ 116us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:46.169509 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:46.169635 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:46.175152 (+ 5517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:46.176497 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:46.176528 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:46.176569 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:46.176752 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:46.177519 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:46.177558 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:46.177581 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:46.178181 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:46.178611 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:46.179716 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0502 14:04:46.180162 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
W20240502 14:04:46.184998 29693 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55716
I20240502 14:04:46.199548 30076 ts_tablet_manager.cc:1888] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:46.200035 30076 log.cc:1196] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/01137234376e47fd8adf03fb9785005a
I20240502 14:04:46.200879 30076 ts_tablet_manager.cc:1909] T 01137234376e47fd8adf03fb9785005a P 685a30641c824069977b69e5ba55a1c4: Deleting consensus metadata
I20240502 14:04:46.202647 30078 ts_tablet_manager.cc:1888] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:46.203066 30078 log.cc:1196] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/01137234376e47fd8adf03fb9785005a
I20240502 14:04:46.203805 30078 ts_tablet_manager.cc:1909] T 01137234376e47fd8adf03fb9785005a P c9cf29676e4f4c4fad94cc354f0aaa84: Deleting consensus metadata
I20240502 14:04:46.203785 29476 catalog_manager.cc:4830] TS 685a30641c824069977b69e5ba55a1c4 (127.27.174.67:36759): tablet 01137234376e47fd8adf03fb9785005a (table test-table [id=3e58edfc5f134864823657b3d845516a]) successfully deleted
I20240502 14:04:46.206255 29474 catalog_manager.cc:4830] TS c9cf29676e4f4c4fad94cc354f0aaa84 (127.27.174.66:37095): tablet 01137234376e47fd8adf03fb9785005a (table test-table [id=3e58edfc5f134864823657b3d845516a]) successfully deleted
I20240502 14:04:46.207535 30077 ts_tablet_manager.cc:1888] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:46.208117 30077 log.cc:1196] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/01137234376e47fd8adf03fb9785005a
I20240502 14:04:46.208925 30077 ts_tablet_manager.cc:1909] T 01137234376e47fd8adf03fb9785005a P 265ad9984f2b4ea8928915e248911683: Deleting consensus metadata
I20240502 14:04:46.211493 29474 catalog_manager.cc:4830] TS 265ad9984f2b4ea8928915e248911683 (127.27.174.65:39075): tablet 01137234376e47fd8adf03fb9785005a (table test-table [id=3e58edfc5f134864823657b3d845516a]) successfully deleted
I20240502 14:04:46.228001 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:46.199105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.27.174.126:38089)
0502 14:04:46.211255 (+ 12150us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:46.211276 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:46.211841 (+ 565us) server_negotiation.cc:407] Connection header received
0502 14:04:46.211979 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:46.212001 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:46.212306 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:46.212451 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:46.214741 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:46.215667 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:46.216798 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:46.217336 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:46.219671 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:46.219735 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:46.219750 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:46.219833 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:46.223210 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:46.224183 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:46.224204 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:46.224222 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:46.224350 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:46.224857 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:46.224876 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:46.224889 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:46.225264 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:46.225580 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:46.226139 (+ 559us) server_negotiation.cc:299] Negotiation successful
0502 14:04:46.226570 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11574,"thread_start_us":11339,"threads_started":1}
W20240502 14:04:46.228976 29489 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51608
I20240502 14:04:46.231281 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29560
I20240502 14:04:46.276170 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29702
I20240502 14:04:46.306339 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29844
I20240502 14:04:46.333050 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 29455
2024-05-02T14:04:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (10239 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:04:46 dist-test-slave-dlb8 krb5kdc[30086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:04:46 dist-test-slave-dlb8 krb5kdc[30086](info): set up 2 sockets
May 02 14:04:46 dist-test-slave-dlb8 krb5kdc[30086](info): commencing operation
krb5kdc: starting...
W20240502 14:04:47.311668 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:04:47 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:04:47.508886 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36253
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:47.794834 30100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:47.795404 30100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:47.795948 30100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:47.825037 30100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:04:47.825330 30100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240502 14:04:47.825592 30100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:47.825870 30100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:47.826123 30100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:04:47.826340 30100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:04:47.860201 30100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:36253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:47.861322 30100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:47.862383 30100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:47.877825 30109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:47.878854 30106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:47.879827 30100 server_base.cc:1034] running on GCE node
W20240502 14:04:47.877864 30107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:48.922073 30100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:48.924614 30100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:48.925977 30100 hybrid_clock.cc:648] HybridClock initialized: now 1714658688925915 us; error 73 us; skew 500 ppm
I20240502 14:04:48.926808 30100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:48 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:48.932823 30100 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:04:48.937697 30100 webserver.cc:460] Webserver started at http://127.27.174.126:34425/ using document root <none> and password file <none>
I20240502 14:04:48.938634 30100 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:48.938838 30100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:48.939281 30100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:48.944031 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "06d26386e9524101b619fa9994abae88"
format_stamp: "Formatted at 2024-05-02 14:04:48 on dist-test-slave-dlb8"
server_key: "4b713293ff89a07185b3937a2753e777"
server_key_iv: "8f6166e6d9cc36e38defc5a1c443f3fa"
server_key_version: "encryptionkey@0"
I20240502 14:04:48.945020 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "06d26386e9524101b619fa9994abae88"
format_stamp: "Formatted at 2024-05-02 14:04:48 on dist-test-slave-dlb8"
server_key: "4b713293ff89a07185b3937a2753e777"
server_key_iv: "8f6166e6d9cc36e38defc5a1c443f3fa"
server_key_version: "encryptionkey@0"
I20240502 14:04:48.951745 30100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20240502 14:04:48.956878 30117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:48.957862 30100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:04:48.958230 30100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "06d26386e9524101b619fa9994abae88"
format_stamp: "Formatted at 2024-05-02 14:04:48 on dist-test-slave-dlb8"
server_key: "4b713293ff89a07185b3937a2753e777"
server_key_iv: "8f6166e6d9cc36e38defc5a1c443f3fa"
server_key_version: "encryptionkey@0"
I20240502 14:04:48.958587 30100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:49.014729 30100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:49.019870 30100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:49.020351 30100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:49.095118 30100 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:36253
I20240502 14:04:49.095238 30178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:36253 every 8 connection(s)
I20240502 14:04:49.097606 30100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:04:49.102584 30179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:49.103139 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30100
I20240502 14:04:49.103551 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:04:49.104602 28345 external_mini_cluster.cc:1428] Setting key 615b18b9d5a38a5baf99b9500d79cd5d
I20240502 14:04:49.124265 30179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Bootstrap starting.
I20240502 14:04:49.129143 30179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Neither blocks nor log segments found. Creating new log.
May 02 14:04:49 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:49.130976 30179 log.cc:824] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:49.135964 30179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: No bootstrap required, opened a new log
I20240502 14:04:49.142263 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:49.107102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.27.174.126:36253)
0502 14:04:49.109700 (+ 2598us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:49.109726 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:49.109778 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:04:49.113693 (+ 3915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:49.113734 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:49.114097 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:49.114451 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:49.116998 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:49.118415 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:49.119695 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:49.120269 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:49.131043 (+ 10774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:49.131141 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:49.131207 (+ 66us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:49.131325 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:49.135956 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:49.137421 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:49.137445 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:49.137477 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:49.137662 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:49.138380 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:49.138400 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:49.138414 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:49.138996 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:49.139318 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:49.139952 (+ 634us) server_negotiation.cc:299] Negotiation successful
0502 14:04:49.140350 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2003,"thread_start_us":1540,"threads_started":1}
I20240502 14:04:49.156184 30179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } }
I20240502 14:04:49.156827 30179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:49.157043 30179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d26386e9524101b619fa9994abae88, State: Initialized, Role: FOLLOWER
I20240502 14:04:49.157886 30179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } }
I20240502 14:04:49.158452 30179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:49.158684 30179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:49.158948 30179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:49.162804 30179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } }
I20240502 14:04:49.163429 30179 leader_election.cc:302] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d26386e9524101b619fa9994abae88; no voters:
I20240502 14:04:49.165082 30179 leader_election.cc:288] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:49.165477 30184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:49.167690 30184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [term 1 LEADER]: Becoming Leader. State: Replica: 06d26386e9524101b619fa9994abae88, State: Running, Role: LEADER
I20240502 14:04:49.168524 30184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } }
I20240502 14:04:49.169359 30179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:49.177382 30185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d26386e9524101b619fa9994abae88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } } }
I20240502 14:04:49.178521 30185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:49.178283 30186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d26386e9524101b619fa9994abae88. Latest consensus state: current_term: 1 leader_uuid: "06d26386e9524101b619fa9994abae88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d26386e9524101b619fa9994abae88" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36253 } } }
I20240502 14:04:49.178961 30186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:49.183619 30193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:49.194957 30193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:49.209955 30193 catalog_manager.cc:1347] Generated new cluster ID: de8f6340c7d44624bd767de8f7225cf5
I20240502 14:04:49.210235 30193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:49.231212 30193 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:49.232549 30193 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:49.242794 30193 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Generated new TSK 0
I20240502 14:04:49.243568 30193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:04:49.306118 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36253
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:49.622236 30205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:49.622807 30205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:49.623502 30205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:49.654207 30205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:49.654553 30205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:49.655120 30205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240502 14:04:49.655313 30205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:49.655745 30205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:04:49.690969 30205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:36253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:49.692123 30205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:49.693101 30205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:49.711051 30212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:49.713109 30211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:49.715152 30205 server_base.cc:1034] running on GCE node
W20240502 14:04:49.714504 30214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:50.754696 30205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:50.759985 30205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:50.761405 30205 hybrid_clock.cc:648] HybridClock initialized: now 1714658690761355 us; error 79 us; skew 500 ppm
I20240502 14:04:50.762249 30205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:50 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:50.767704 30205 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:04:50.773578 30205 webserver.cc:460] Webserver started at http://127.27.174.65:36303/ using document root <none> and password file <none>
I20240502 14:04:50.774485 30205 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:50.774665 30205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:50.775063 30205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:50.779888 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "2febb32cb57e4de6836a2050e0cd3046"
format_stamp: "Formatted at 2024-05-02 14:04:50 on dist-test-slave-dlb8"
server_key: "2117bebbae8d8529f6998ce297e84a4f"
server_key_iv: "486e99f0e619f196a98e865fd09c079e"
server_key_version: "encryptionkey@0"
I20240502 14:04:50.780957 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "2febb32cb57e4de6836a2050e0cd3046"
format_stamp: "Formatted at 2024-05-02 14:04:50 on dist-test-slave-dlb8"
server_key: "2117bebbae8d8529f6998ce297e84a4f"
server_key_iv: "486e99f0e619f196a98e865fd09c079e"
server_key_version: "encryptionkey@0"
I20240502 14:04:50.788372 30205 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240502 14:04:50.794173 30222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:50.795195 30205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:04:50.795481 30205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "2febb32cb57e4de6836a2050e0cd3046"
format_stamp: "Formatted at 2024-05-02 14:04:50 on dist-test-slave-dlb8"
server_key: "2117bebbae8d8529f6998ce297e84a4f"
server_key_iv: "486e99f0e619f196a98e865fd09c079e"
server_key_version: "encryptionkey@0"
I20240502 14:04:50.795792 30205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:50.843693 30205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:50.849251 30205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:50.849699 30205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:50.865691 30205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:50.866214 30205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20240502 14:04:50.866662 30205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:50.867004 30205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:50 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:50.901646 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:50.857414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:50029 (local address 127.27.174.126:36253)
0502 14:04:50.871314 (+ 13900us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:50.871337 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:50.873092 (+ 1755us) server_negotiation.cc:407] Connection header received
0502 14:04:50.878244 (+ 5152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:50.878282 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:50.878566 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:50.878796 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:50.881944 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.883185 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:50.884879 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.885431 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:50.890003 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:50.890118 (+ 115us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:50.890136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:50.890239 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:50.893764 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:50.894927 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:50.894969 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:50.895002 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:50.895249 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:50.896226 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:50.896276 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:50.896309 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:50.896889 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:50.897132 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:50.898592 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0502 14:04:50.899253 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13231,"thread_start_us":12997,"threads_started":1}
I20240502 14:04:50.902669 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:50.857722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.65:50029)
0502 14:04:50.872782 (+ 15060us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:50.872877 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:50.877790 (+ 4913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:50.879316 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:50.879357 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:50.880518 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:50.881621 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:50.881662 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.883494 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:50.883524 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:50.884645 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:50.884687 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.885047 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:50.886100 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:50.886175 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:50.889597 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:50.893986 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:50.894009 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:50.894031 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:50.894723 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:50.895541 (+ 818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:50.895588 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:50.895624 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:50.895958 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:50.897138 (+ 1180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:50.897183 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:50.897795 (+ 612us) client_negotiation.cc:769] Sending connection context
0502 14:04:50.898398 (+ 603us) client_negotiation.cc:240] Negotiation successful
0502 14:04:50.899175 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14336,"thread_start_us":13861,"threads_started":1}
I20240502 14:04:50.939921 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:50.917347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33845 (local address 127.27.174.126:36253)
0502 14:04:50.917825 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:50.917843 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:50.921051 (+ 3208us) server_negotiation.cc:407] Connection header received
0502 14:04:50.921381 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:50.921405 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:50.921565 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:50.921702 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:50.924213 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.925087 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:50.926915 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.927876 (+ 961us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:50.929467 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:50.929550 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:50.929566 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:50.929652 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:50.932760 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:50.934026 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:50.934101 (+ 75us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:50.934137 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:50.934334 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:50.935084 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:50.935121 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:50.935152 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:50.935725 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:50.935992 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:50.936927 (+ 935us) server_negotiation.cc:299] Negotiation successful
0502 14:04:50.937490 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20240502 14:04:50.940052 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:50.917252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.65:33845)
0502 14:04:50.920874 (+ 3622us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:50.920917 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:50.921179 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:50.921795 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:50.921819 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:50.922732 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:50.923901 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:50.923965 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.925345 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:50.925363 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:50.926691 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:50.926723 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:50.927010 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:50.927898 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:50.928012 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:50.929209 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:50.932945 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:50.932976 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:50.932998 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:50.933709 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:50.934532 (+ 823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:50.934550 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:50.934563 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:50.934823 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:50.935967 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:50.936006 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:50.936381 (+ 375us) client_negotiation.cc:769] Sending connection context
0502 14:04:50.936700 (+ 319us) client_negotiation.cc:240] Negotiation successful
0502 14:04:50.937254 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3114,"thread_start_us":2235,"threads_started":1}
I20240502 14:04:51.017867 30205 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:33865
I20240502 14:04:51.018481 30342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:33865 every 8 connection(s)
I20240502 14:04:51.020321 30205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:04:51.024402 30343 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36253
I20240502 14:04:51.024839 30343 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:51.025691 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30205
I20240502 14:04:51.025768 30343 heartbeater.cc:511] Master 127.27.174.126:36253 requested a full tablet report, sending...
I20240502 14:04:51.026072 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:04:51.027225 28345 external_mini_cluster.cc:1428] Setting key 0b3d949184a7af03dcb3a6c8bdc26065
I20240502 14:04:51.028591 30134 ts_manager.cc:194] Registered new tserver with Master: 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865)
I20240502 14:04:51.030932 30134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:50029
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:04:51.072691 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36253
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:51.356562 30347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:51.357100 30347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:51.357767 30347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:51.387006 30347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:51.387394 30347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:51.387992 30347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240502 14:04:51.388224 30347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:51.388696 30347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:04:51.423620 30347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:36253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:51.424712 30347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:51.425714 30347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:51.440991 30353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:52.035236 30343 heartbeater.cc:503] Master 127.27.174.126:36253 was elected leader, sending a full tablet report...
W20240502 14:04:51.441037 30354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:51.443656 30356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:51.442734 30347 server_base.cc:1034] running on GCE node
I20240502 14:04:52.493748 30347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:52.496011 30347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:52.497346 30347 hybrid_clock.cc:648] HybridClock initialized: now 1714658692497323 us; error 43 us; skew 500 ppm
I20240502 14:04:52.498077 30347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:52 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:52.503793 30347 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:04:52.508160 30347 webserver.cc:460] Webserver started at http://127.27.174.66:43955/ using document root <none> and password file <none>
I20240502 14:04:52.509038 30347 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:52.509241 30347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:52.509668 30347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:52.514675 30347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "25cad6faa1e745fbba1db358f5313861"
format_stamp: "Formatted at 2024-05-02 14:04:52 on dist-test-slave-dlb8"
server_key: "f6ab71453c4e58529eb0d5cde3c7825d"
server_key_iv: "310776df3ad6fa24295d975a2029db3c"
server_key_version: "encryptionkey@0"
I20240502 14:04:52.515728 30347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "25cad6faa1e745fbba1db358f5313861"
format_stamp: "Formatted at 2024-05-02 14:04:52 on dist-test-slave-dlb8"
server_key: "f6ab71453c4e58529eb0d5cde3c7825d"
server_key_iv: "310776df3ad6fa24295d975a2029db3c"
server_key_version: "encryptionkey@0"
I20240502 14:04:52.522382 30347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:04:52.527559 30364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:52.528650 30347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20240502 14:04:52.528957 30347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "25cad6faa1e745fbba1db358f5313861"
format_stamp: "Formatted at 2024-05-02 14:04:52 on dist-test-slave-dlb8"
server_key: "f6ab71453c4e58529eb0d5cde3c7825d"
server_key_iv: "310776df3ad6fa24295d975a2029db3c"
server_key_version: "encryptionkey@0"
I20240502 14:04:52.529304 30347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:52.581880 30347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:52.586982 30347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:52.587414 30347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:52.596431 30347 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:52.596832 30347 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:04:52.597190 30347 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:52.597445 30347 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:52 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:52.627041 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:52.593667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:51861 (local address 127.27.174.126:36253)
0502 14:04:52.597344 (+ 3677us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:52.597367 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:52.597433 (+ 66us) server_negotiation.cc:407] Connection header received
0502 14:04:52.602979 (+ 5546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:52.603014 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:52.603284 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:52.603508 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:52.605915 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.607418 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:52.609658 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.610333 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:52.614900 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:52.614975 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:52.614992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:52.615088 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:52.620675 (+ 5587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:52.621834 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:52.621857 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:52.621871 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:52.622008 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:52.622712 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:52.622734 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:52.622748 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:52.623143 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:52.623383 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:52.624382 (+ 999us) server_negotiation.cc:299] Negotiation successful
0502 14:04:52.624843 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3049,"thread_start_us":1715,"threads_started":1}
I20240502 14:04:52.627885 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:52.593864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.66:51861)
0502 14:04:52.597190 (+ 3326us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:52.597281 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:52.602589 (+ 5308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:52.603686 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:52.603712 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:52.604548 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:52.605590 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:52.605633 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.607725 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:52.607763 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:52.609402 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:52.609444 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.609732 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:52.610842 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:52.610932 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:52.614619 (+ 3687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:52.620938 (+ 6319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:52.620968 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:52.620991 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:52.621621 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:52.622213 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:52.622250 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:52.622279 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:52.622525 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:52.623368 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:52.623400 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:52.623823 (+ 423us) client_negotiation.cc:769] Sending connection context
0502 14:04:52.624194 (+ 371us) client_negotiation.cc:240] Negotiation successful
0502 14:04:52.624846 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2659,"thread_start_us":2030,"threads_started":1}
I20240502 14:04:52.666500 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:52.643567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58393 (local address 127.27.174.126:36253)
0502 14:04:52.644345 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:52.644378 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:52.647064 (+ 2686us) server_negotiation.cc:407] Connection header received
0502 14:04:52.647439 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:52.647463 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:52.647634 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:52.647797 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:52.650361 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.651841 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:52.654125 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.654782 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:52.656833 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:52.656904 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:52.656921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:52.657014 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:52.660763 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:52.661851 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:52.661871 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:52.661887 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:52.662026 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:52.662604 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:52.662622 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:52.662636 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:52.663069 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:52.663292 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:52.663984 (+ 692us) server_negotiation.cc:299] Negotiation successful
0502 14:04:52.664429 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20240502 14:04:52.666889 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:52.643365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.66:58393)
0502 14:04:52.646866 (+ 3501us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:52.646915 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:52.647233 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:52.647892 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:52.647919 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:52.648805 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:52.649938 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:52.649993 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.652176 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:52.652218 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:52.653818 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:52.653879 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:52.654278 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:52.655338 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:52.655432 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:52.656603 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:52.660954 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:52.660981 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:52.661000 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:52.661644 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:52.662220 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:52.662238 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:52.662261 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:52.662420 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:52.663240 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:52.663266 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:52.663527 (+ 261us) client_negotiation.cc:769] Sending connection context
0502 14:04:52.663750 (+ 223us) client_negotiation.cc:240] Negotiation successful
0502 14:04:52.664244 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3046,"thread_start_us":1840,"threads_started":1}
I20240502 14:04:52.749564 30347 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:42735
I20240502 14:04:52.750152 30484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:42735 every 8 connection(s)
I20240502 14:04:52.751986 30347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:04:52.755519 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30347
I20240502 14:04:52.756036 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:04:52.756402 30485 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36253
I20240502 14:04:52.756817 30485 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:52.757316 28345 external_mini_cluster.cc:1428] Setting key dc815b6f16647278b49affe7c9eda877
I20240502 14:04:52.758141 30485 heartbeater.cc:511] Master 127.27.174.126:36253 requested a full tablet report, sending...
I20240502 14:04:52.760385 30133 ts_manager.cc:194] Registered new tserver with Master: 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:52.761826 30133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:51861
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:04:52.802178 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36253
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:04:53.049913 30339 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc060 after lost signal to thread 30206
W20240502 14:04:53.050729 30339 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcfe0 after lost signal to thread 30342
W20240502 14:04:53.092182 30489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:04:53.092924 30489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:04:53.093788 30489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:04:53.124209 30489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:04:53.124562 30489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:04:53.125171 30489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240502 14:04:53.125381 30489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:04:53.125849 30489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:04:53.159116 30489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:36253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:04:53.160261 30489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:53.161226 30489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:53.179467 30495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:53.765247 30485 heartbeater.cc:503] Master 127.27.174.126:36253 was elected leader, sending a full tablet report...
W20240502 14:04:53.180255 30496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:53.181186 30489 server_base.cc:1034] running on GCE node
W20240502 14:04:53.180667 30498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:54.265483 30489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:04:54.268251 30489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:04:54.269682 30489 hybrid_clock.cc:648] HybridClock initialized: now 1714658694269627 us; error 78 us; skew 500 ppm
I20240502 14:04:54.270466 30489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:04:54 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:04:54.280678 30489 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:04:54.286537 30489 webserver.cc:460] Webserver started at http://127.27.174.67:35693/ using document root <none> and password file <none>
I20240502 14:04:54.287376 30489 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:54.287580 30489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:54.287978 30489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:54.293341 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "fdd4b183fbd747dd89385c76fb97f2f2"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-dlb8"
server_key: "f94e9e0d342628aeb40848e0fb4c159d"
server_key_iv: "32c28c5ec4849e4f7fab7322e92ac643"
server_key_version: "encryptionkey@0"
I20240502 14:04:54.294428 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "fdd4b183fbd747dd89385c76fb97f2f2"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-dlb8"
server_key: "f94e9e0d342628aeb40848e0fb4c159d"
server_key_iv: "32c28c5ec4849e4f7fab7322e92ac643"
server_key_version: "encryptionkey@0"
I20240502 14:04:54.301620 30489 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20240502 14:04:54.307703 30506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.308785 30489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:04:54.309087 30489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "fdd4b183fbd747dd89385c76fb97f2f2"
format_stamp: "Formatted at 2024-05-02 14:04:54 on dist-test-slave-dlb8"
server_key: "f94e9e0d342628aeb40848e0fb4c159d"
server_key_iv: "32c28c5ec4849e4f7fab7322e92ac643"
server_key_version: "encryptionkey@0"
I20240502 14:04:54.309430 30489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:54.368914 30489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:54.374382 30489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:54.374817 30489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:54.391651 30489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:54.392040 30489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:54.392500 30489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:54.392716 30489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:04:54 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:54.437083 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.382868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.67:48823)
0502 14:04:54.396902 (+ 14034us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:54.396996 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:54.404534 (+ 7538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:54.406742 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:54.406809 (+ 67us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:54.408226 (+ 1417us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:54.409855 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.409920 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.412242 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.412273 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:54.413671 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.413729 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.414149 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.415271 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:54.415374 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:54.420069 (+ 4695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:54.428185 (+ 8116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:54.428238 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:54.428286 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:54.429454 (+ 1168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:54.430629 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:54.430662 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:54.430692 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:54.431026 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:54.432295 (+ 1269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:54.432333 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:54.432918 (+ 585us) client_negotiation.cc:769] Sending connection context
0502 14:04:54.433322 (+ 404us) client_negotiation.cc:240] Negotiation successful
0502 14:04:54.433976 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13282,"thread_start_us":8284,"threads_started":1}
I20240502 14:04:54.439953 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.382608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:48823 (local address 127.27.174.126:36253)
0502 14:04:54.393436 (+ 10828us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.393459 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.397235 (+ 3776us) server_negotiation.cc:407] Connection header received
0502 14:04:54.405140 (+ 7905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.405169 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.405517 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.405675 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:54.410313 (+ 4638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.411869 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.414782 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.415487 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.421520 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:54.421634 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:54.421670 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:54.421838 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:54.427859 (+ 6021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.430128 (+ 2269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.430163 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.430191 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.430401 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.431438 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.431471 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.431507 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.432032 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:54.436156 (+ 4124us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.436542 (+ 386us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.437149 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10119,"thread_start_us":9781,"threads_started":1}
I20240502 14:04:54.491595 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.461196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.27.174.67:52257)
0502 14:04:54.465651 (+ 4455us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:54.465720 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:54.466073 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:54.467177 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:54.467216 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:54.468287 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:54.469435 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.469587 (+ 152us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.471636 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.471666 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:54.472991 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.473036 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.473355 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.474305 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:54.474388 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:54.475954 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:54.482531 (+ 6577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:54.482566 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:54.482586 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:54.483442 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:54.484298 (+ 856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:54.484341 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:54.484376 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:54.484603 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:54.485750 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:54.485780 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:54.486120 (+ 340us) client_negotiation.cc:769] Sending connection context
0502 14:04:54.486435 (+ 315us) client_negotiation.cc:240] Negotiation successful
0502 14:04:54.486950 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3670,"thread_start_us":3143,"threads_started":1}
I20240502 14:04:54.492307 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.461482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52257 (local address 127.27.174.126:36253)
0502 14:04:54.462627 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.462675 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.465883 (+ 3208us) server_negotiation.cc:407] Connection header received
0502 14:04:54.466416 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.466454 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.466742 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.466905 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:54.469844 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.471256 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.473292 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.473975 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.476245 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:54.476369 (+ 124us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:54.476405 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:54.476561 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:54.482216 (+ 5655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.483795 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.483835 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.483868 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.484052 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.484891 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.484929 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.484960 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.485535 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:54.485794 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.489780 (+ 3986us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.490308 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20240502 14:04:54.584167 30489 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41667
I20240502 14:04:54.584885 30626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41667 every 8 connection(s)
I20240502 14:04:54.586634 30489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:04:54.591856 30627 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36253
I20240502 14:04:54.592285 30627 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:54.593614 30627 heartbeater.cc:511] Master 127.27.174.126:36253 requested a full tablet report, sending...
I20240502 14:04:54.593698 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30489
I20240502 14:04:54.594126 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:04:54.595160 28345 external_mini_cluster.cc:1428] Setting key d364b4271e0c02849e2262cad1663fb7
I20240502 14:04:54.596283 30133 ts_manager.cc:194] Registered new tserver with Master: fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667)
I20240502 14:04:54.597777 30133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:48823
I20240502 14:04:54.600555 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:04:54 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 02 14:04:54 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:54.671550 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.647564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.27.174.126:36253)
0502 14:04:54.648352 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.648395 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.649331 (+ 936us) server_negotiation.cc:407] Connection header received
0502 14:04:54.649752 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.649768 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.649924 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.650099 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:54.653337 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.654243 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.656204 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.656540 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.662689 (+ 6149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:54.662766 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:54.662783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:54.662887 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:54.666317 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.667609 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.667628 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.667642 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.667779 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:54.668349 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:54.668366 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:54.668379 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:54.668758 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:54.669029 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.669737 (+ 708us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.670183 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20240502 14:04:54.677222 30133 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:54.700225 30133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:54.755242 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.729607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:42735 (local address 127.0.0.1:41620)
0502 14:04:54.747225 (+ 17618us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:54.747308 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:54.747764 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:54.749207 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:54.749237 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:54.749346 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:54.749994 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.750033 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.751687 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.751706 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:54.752881 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.752921 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.753227 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.753375 (+ 148us) client_negotiation.cc:769] Sending connection context
0502 14:04:54.753541 (+ 166us) client_negotiation.cc:240] Negotiation successful
0502 14:04:54.753766 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17030,"thread_start_us":16742,"threads_started":1}
I20240502 14:04:54.759052 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.732381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.27.174.66:42735)
0502 14:04:54.744861 (+ 12480us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.744903 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.747464 (+ 2561us) server_negotiation.cc:407] Connection header received
0502 14:04:54.748103 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.748133 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.748527 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.748720 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:54.750257 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.751448 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.753144 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.754332 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.756990 (+ 2658us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.757188 (+ 198us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.757439 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11625,"thread_start_us":12419,"threads_started":1}
I20240502 14:04:54.763468 30420 tablet_service.cc:1467] Processing CreateTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb (DEFAULT_TABLE table=test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:54.765319 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573ea7ce02a4f4f8459334f188d9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:54.767637 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.731291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41667 (local address 127.0.0.1:57814)
0502 14:04:54.756432 (+ 25141us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:54.756474 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:54.756682 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:54.760027 (+ 3345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:54.760059 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:54.760162 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:54.760641 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.760690 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.763835 (+ 3145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.763866 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:54.765506 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.765563 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.765846 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.765929 (+ 83us) client_negotiation.cc:769] Sending connection context
0502 14:04:54.766065 (+ 136us) client_negotiation.cc:240] Negotiation successful
0502 14:04:54.766264 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24596,"thread_start_us":25503,"threads_started":1}
I20240502 14:04:54.771808 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.733963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.27.174.67:41667)
0502 14:04:54.758455 (+ 24492us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.758488 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.758543 (+ 55us) server_negotiation.cc:407] Connection header received
0502 14:04:54.758734 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.758764 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.759117 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.759603 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:54.761333 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.763560 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.768267 (+ 4707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.769659 (+ 1392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.769875 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.770091 (+ 216us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.770377 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23788,"thread_start_us":25072,"threads_started":1}
I20240502 14:04:54.776216 30562 tablet_service.cc:1467] Processing CreateTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb (DEFAULT_TABLE table=test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:54.778152 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573ea7ce02a4f4f8459334f188d9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:54.787283 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.757850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.27.174.65:33865)
0502 14:04:54.774710 (+ 16860us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:54.774742 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:54.774794 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:04:54.775035 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:54.775070 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:54.775408 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:54.775624 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:54.777375 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.779250 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.783972 (+ 4722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.785245 (+ 1273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.785461 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:54.785658 (+ 197us) server_negotiation.cc:299] Negotiation successful
0502 14:04:54.785887 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16242,"thread_start_us":15883,"threads_started":1}
I20240502 14:04:54.787880 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:54.763913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.0.0.1:47458)
0502 14:04:54.765018 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:54.765064 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:54.765320 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:54.776022 (+ 10702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:54.776053 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:54.776185 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:54.777009 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.777062 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.779547 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:54.779577 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:54.781304 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:54.781354 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:54.781612 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:54.781690 (+ 78us) client_negotiation.cc:769] Sending connection context
0502 14:04:54.781844 (+ 154us) client_negotiation.cc:240] Negotiation successful
0502 14:04:54.782074 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20240502 14:04:54.792727 30278 tablet_service.cc:1467] Processing CreateTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb (DEFAULT_TABLE table=test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:54.794612 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573ea7ce02a4f4f8459334f188d9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:54.819423 30645 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Bootstrap starting.
I20240502 14:04:54.826012 30645 tablet_bootstrap.cc:654] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:54.826640 30646 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Bootstrap starting.
I20240502 14:04:54.829463 30645 log.cc:824] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:54.832726 30647 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Bootstrap starting.
I20240502 14:04:54.833631 30646 tablet_bootstrap.cc:654] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:54.835304 30646 log.cc:824] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:54.838887 30645 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: No bootstrap required, opened a new log
I20240502 14:04:54.838954 30647 tablet_bootstrap.cc:654] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:54.839407 30645 ts_tablet_manager.cc:1367] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.007s
I20240502 14:04:54.841429 30647 log.cc:824] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:54.841527 30646 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: No bootstrap required, opened a new log
I20240502 14:04:54.842172 30646 ts_tablet_manager.cc:1367] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20240502 14:04:54.847313 30647 tablet_bootstrap.cc:492] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: No bootstrap required, opened a new log
I20240502 14:04:54.847805 30647 ts_tablet_manager.cc:1367] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 14:04:54.863596 30646 raft_consensus.cc:348] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.864284 30646 raft_consensus.cc:374] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:54.864535 30646 raft_consensus.cc:729] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Initialized, Role: FOLLOWER
I20240502 14:04:54.865260 30646 consensus_queue.cc:260] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.868290 30627 heartbeater.cc:503] Master 127.27.174.126:36253 was elected leader, sending a full tablet report...
I20240502 14:04:54.868969 30646 ts_tablet_manager.cc:1398] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Time spent starting tablet: real 0.027s user 0.021s sys 0.005s
I20240502 14:04:54.869397 30645 raft_consensus.cc:348] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.870292 30645 raft_consensus.cc:374] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:54.870571 30645 raft_consensus.cc:729] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Initialized, Role: FOLLOWER
I20240502 14:04:54.871299 30645 consensus_queue.cc:260] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.876974 30645 ts_tablet_manager.cc:1398] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Time spent starting tablet: real 0.037s user 0.029s sys 0.009s
I20240502 14:04:54.879190 30647 raft_consensus.cc:348] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.880110 30647 raft_consensus.cc:374] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:54.880424 30647 raft_consensus.cc:729] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febb32cb57e4de6836a2050e0cd3046, State: Initialized, Role: FOLLOWER
I20240502 14:04:54.881326 30647 consensus_queue.cc:260] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:54.884634 30647 ts_tablet_manager.cc:1398] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Time spent starting tablet: real 0.037s user 0.028s sys 0.007s
W20240502 14:04:54.989418 30481 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc0e0 after lost signal to thread 30348
W20240502 14:04:55.008474 30486 tablet.cc:2369] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:04:55.028661 30344 tablet.cc:2369] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:04:55.092229 30628 tablet.cc:2369] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:55.115088 30651 raft_consensus.cc:482] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.115526 30651 raft_consensus.cc:504] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:55.117671 30651 leader_election.cc:288] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735), 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865)
I20240502 14:04:55.141083 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.119544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56509 (local address 127.27.174.66:42735)
0502 14:04:55.120227 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.120251 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.131285 (+ 11034us) server_negotiation.cc:407] Connection header received
0502 14:04:55.132041 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.132083 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.132347 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.132483 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:55.133969 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.135210 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.137486 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.138666 (+ 1180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.138806 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.139480 (+ 674us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.139712 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20240502 14:04:55.144003 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.117961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:42735 (local address 127.27.174.67:56509)
0502 14:04:55.131042 (+ 13081us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.131120 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.131659 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.132904 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.132931 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.133026 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:55.133680 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.133727 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.135525 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.135560 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.137186 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.137237 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.139054 (+ 1817us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.139138 (+ 84us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.142120 (+ 2982us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.142323 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11716,"thread_start_us":12407,"threads_started":1}
I20240502 14:04:55.144299 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.120142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44123 (local address 127.27.174.65:33865)
0502 14:04:55.121005 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.121038 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.133433 (+ 12395us) server_negotiation.cc:407] Connection header received
0502 14:04:55.135264 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.135295 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.135545 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.135675 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:55.137055 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.138251 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.140777 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.141905 (+ 1128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.142044 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.142823 (+ 779us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.143071 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20240502 14:04:55.146147 30440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573ea7ce02a4f4f8459334f188d9bdb" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861" is_pre_election: true
I20240502 14:04:55.146848 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.118698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.27.174.67:44123)
0502 14:04:55.133246 (+ 14548us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.133305 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.134025 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.136032 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.136058 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.136153 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:55.136765 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.136816 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.138603 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.138642 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.140426 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.140477 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.142413 (+ 1936us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.142494 (+ 81us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.144999 (+ 2505us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.145247 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13967,"thread_start_us":14659,"threads_started":1}
I20240502 14:04:55.147055 30440 raft_consensus.cc:2454] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:55.148409 30510 leader_election.cc:302] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:55.148602 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573ea7ce02a4f4f8459334f188d9bdb" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:55.149319 30651 raft_consensus.cc:2781] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.149662 30651 raft_consensus.cc:482] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.149533 30298 raft_consensus.cc:2454] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:55.149950 30651 raft_consensus.cc:3037] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.156461 30651 raft_consensus.cc:504] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:55.159231 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573ea7ce02a4f4f8459334f188d9bdb" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046"
I20240502 14:04:55.159199 30440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573ea7ce02a4f4f8459334f188d9bdb" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861"
I20240502 14:04:55.159767 30298 raft_consensus.cc:3037] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.159868 30440 raft_consensus.cc:3037] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.161294 30651 leader_election.cc:288] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Requested vote from peers 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735), 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865)
I20240502 14:04:55.166241 30298 raft_consensus.cc:2454] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:55.167325 30507 leader_election.cc:302] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:55.167662 30440 raft_consensus.cc:2454] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:55.168088 30651 raft_consensus.cc:2781] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.169776 30651 raft_consensus.cc:686] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Becoming Leader. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Running, Role: LEADER
I20240502 14:04:55.171303 30651 consensus_queue.cc:237] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } }
I20240502 14:04:55.185050 30134 catalog_manager.cc:5484] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } health_report { overall_health: UNKNOWN } } }
May 02 14:04:55 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:55.287868 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.263621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36253 (local address 127.0.0.1:54180)
0502 14:04:55.267016 (+ 3395us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.267052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.267267 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.267945 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.267962 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.268945 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:04:55.269344 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.269376 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.271480 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.271501 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.272310 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.272345 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.272583 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.273340 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:04:55.273472 (+ 132us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:04:55.276302 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:04:55.282800 (+ 6498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:55.282828 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:55.282843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:55.283482 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:55.284126 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:04:55.284148 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:04:55.284164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:04:55.284332 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:04:55.285302 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:04:55.285322 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:04:55.285574 (+ 252us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.285794 (+ 220us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.286207 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2947,"spinlock_wait_cycles":2304,"thread_start_us":2622,"threads_started":1}
I20240502 14:04:55.289234 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.263694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.27.174.126:36253)
0502 14:04:55.266655 (+ 2961us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.266682 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.267170 (+ 488us) server_negotiation.cc:407] Connection header received
0502 14:04:55.267455 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.267485 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.267705 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.267893 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:55.269631 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.271273 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.272553 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.273151 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.276570 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:55.276699 (+ 129us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:55.276730 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:55.276867 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:55.282590 (+ 5723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:55.283667 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:55.283703 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:55.283728 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:55.283920 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:55.284494 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:55.284528 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:55.284553 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:55.285109 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:55.285374 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.286068 (+ 694us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.286589 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2238,"thread_start_us":1973,"threads_started":1}
I20240502 14:04:55.297757 30134 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54180:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240502 14:04:55.300781 30134 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:55.320242 30278 tablet_service.cc:1467] Processing CreateTablet for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73e872dc075c443d9f7528f94f5c8792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:55.320804 30420 tablet_service.cc:1467] Processing CreateTablet for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73e872dc075c443d9f7528f94f5c8792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:55.321946 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.322361 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.321906 30562 tablet_service.cc:1467] Processing CreateTablet for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73e872dc075c443d9f7528f94f5c8792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:04:55.323369 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ea78345ebe4e1aaba5e6e60f6ba85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.339524 30647 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: Bootstrap starting.
I20240502 14:04:55.340952 30646 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: Bootstrap starting.
I20240502 14:04:55.344586 30645 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: Bootstrap starting.
I20240502 14:04:55.346467 30647 tablet_bootstrap.cc:654] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.346931 30646 tablet_bootstrap.cc:654] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.351451 30645 tablet_bootstrap.cc:654] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:55.357754 30647 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: No bootstrap required, opened a new log
I20240502 14:04:55.359850 30645 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: No bootstrap required, opened a new log
I20240502 14:04:55.360431 30645 ts_tablet_manager.cc:1367] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240502 14:04:55.358260 30647 ts_tablet_manager.cc:1367] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20240502 14:04:55.363093 30647 raft_consensus.cc:348] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.363898 30647 raft_consensus.cc:374] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.364223 30647 raft_consensus.cc:729] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febb32cb57e4de6836a2050e0cd3046, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.364493 30646 tablet_bootstrap.cc:492] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: No bootstrap required, opened a new log
I20240502 14:04:55.364143 30645 raft_consensus.cc:348] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.364941 30646 ts_tablet_manager.cc:1367] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.008s
I20240502 14:04:55.364838 30645 raft_consensus.cc:374] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.365239 30645 raft_consensus.cc:729] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.365051 30647 consensus_queue.cc:260] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.365995 30645 consensus_queue.cc:260] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.366418 30653 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:55.367233 30653 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:55.367291 30646 raft_consensus.cc:348] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.367334 30652 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:55.367952 30646 raft_consensus.cc:374] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:55.368091 30652 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:55.368211 30646 raft_consensus.cc:729] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Initialized, Role: FOLLOWER
I20240502 14:04:55.368921 30646 consensus_queue.cc:260] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.370126 30651 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:55.370841 30651 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:55.371045 30646 ts_tablet_manager.cc:1398] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 14:04:55.369596 30645 ts_tablet_manager.cc:1398] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240502 14:04:55.376142 30647 ts_tablet_manager.cc:1398] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 14:04:55.679756 30651 consensus_queue.cc:1035] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:55.685758 30652 raft_consensus.cc:482] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.686223 30652 raft_consensus.cc:504] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.688979 30652 leader_election.cc:288] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667)
I20240502 14:04:55.712728 30651 raft_consensus.cc:482] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:55.713263 30651 raft_consensus.cc:504] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.715999 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:55.716666 30298 raft_consensus.cc:2454] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:55.717657 30507 leader_election.cc:302] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:55.719131 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.691221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:43801 (local address 127.27.174.65:33865)
0502 14:04:55.694148 (+ 2927us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.694173 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.702546 (+ 8373us) server_negotiation.cc:407] Connection header received
0502 14:04:55.708489 (+ 5943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.708538 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.708865 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.709042 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:55.710881 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.712631 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.715399 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.716596 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.716737 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.717369 (+ 632us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.717591 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2238,"thread_start_us":2118,"threads_started":1}
I20240502 14:04:55.721707 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.690178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.27.174.66:43801)
0502 14:04:55.702363 (+ 12185us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.702422 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.702685 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.709673 (+ 6988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.709733 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.709877 (+ 144us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:55.710557 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.710608 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.712942 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.712971 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.715091 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.715139 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.716976 (+ 1837us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.717063 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.719896 (+ 2833us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.720098 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11468,"thread_start_us":11141,"threads_started":1}
I20240502 14:04:55.722599 30440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861" is_pre_election: true
I20240502 14:04:55.723222 30440 raft_consensus.cc:2454] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:55.724138 30651 leader_election.cc:288] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:55.724586 30651 raft_consensus.cc:2781] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:55.724820 30651 raft_consensus.cc:482] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:55.725045 30651 raft_consensus.cc:3037] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.729624 30651 raft_consensus.cc:504] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.739037 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046"
I20240502 14:04:55.739923 30298 raft_consensus.cc:3037] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.750044 30440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861"
I20240502 14:04:55.750845 30440 raft_consensus.cc:3037] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:55.759476 30440 raft_consensus.cc:2454] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:55.761142 30510 leader_election.cc:302] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:55.768366 30297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:55.771633 30676 raft_consensus.cc:2781] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:55.775574 30677 consensus_queue.cc:1035] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:04:55.776942 30651 leader_election.cc:288] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Requested vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:55.788601 30676 raft_consensus.cc:686] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Becoming Leader. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Running, Role: LEADER
I20240502 14:04:55.792029 30298 raft_consensus.cc:2454] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:55.789462 30676 consensus_queue.cc:237] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:55.798940 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.710499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41667 (local address 127.27.174.66:53517)
0502 14:04:55.722836 (+ 12337us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.722880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.723165 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.789737 (+ 66572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.789765 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.789884 (+ 119us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:04:55.790431 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.790506 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.794723 (+ 4217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.794765 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.796322 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.796374 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.796663 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.796748 (+ 85us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.796875 (+ 127us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.797073 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11794,"thread_start_us":55705,"threads_started":1}
I20240502 14:04:55.803897 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.702548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53517 (local address 127.27.174.67:41667)
0502 14:04:55.788695 (+ 86147us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.788720 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.788779 (+ 59us) server_negotiation.cc:407] Connection header received
0502 14:04:55.789016 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.789046 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.789384 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.791686 (+ 2302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:04:55.792114 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.794441 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.800202 (+ 5761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.801275 (+ 1073us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.801409 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.801619 (+ 210us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.801887 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85413,"thread_start_us":85143,"threads_started":1}
I20240502 14:04:55.805166 30677 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: TxnStatusTablet state changed. Reason: New leader fdd4b183fbd747dd89385c76fb97f2f2. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:55.805917 30677 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:55.807930 30133 catalog_manager.cc:5484] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:55.810704 30582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ea78345ebe4e1aaba5e6e60f6ba85f" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" is_pre_election: true
I20240502 14:04:55.812909 30365 leader_election.cc:302] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861; no voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2
I20240502 14:04:55.814131 30652 raft_consensus.cc:2726] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:55.821198 30682 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240502 14:04:55.821885 30682 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240502 14:04:55.960317 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.949022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.27.174.67:41667)
0502 14:04:55.949757 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:55.949784 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:55.951131 (+ 1347us) server_negotiation.cc:407] Connection header received
0502 14:04:55.951631 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:55.951666 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:55.952034 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:55.952242 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:55.953248 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.954634 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.956551 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.957118 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.957320 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:55.958027 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:55.958247 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:55.958523 (+ 276us) server_negotiation.cc:299] Negotiation successful
0502 14:04:55.958709 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20240502 14:04:55.960767 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:55.948864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41667 (local address 127.0.0.1:57828)
0502 14:04:55.950954 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:55.951001 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:55.951322 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:55.952306 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:55.952335 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:55.952388 (+ 53us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:55.952962 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.953015 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.954905 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:55.954940 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:55.956284 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:55.956339 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:55.956678 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:55.956802 (+ 124us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:55.958231 (+ 1429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:55.958280 (+ 49us) client_negotiation.cc:769] Sending connection context
0502 14:04:55.958451 (+ 171us) client_negotiation.cc:240] Negotiation successful
0502 14:04:55.958711 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":1133,"threads_started":1}
I20240502 14:04:55.974124 30439 raft_consensus.cc:1261] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:55.974934 30298 raft_consensus.cc:1261] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:55.975891 30677 consensus_queue.cc:1035] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:55.976603 30669 consensus_queue.cc:1035] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.001917 30653 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: TxnStatusTablet state changed. Reason: New leader fdd4b183fbd747dd89385c76fb97f2f2. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.001973 30652 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: TxnStatusTablet state changed. Reason: New leader fdd4b183fbd747dd89385c76fb97f2f2. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.002665 30653 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:56.002801 30652 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:56.008728 30672 mvcc.cc:205] Tried to move back new op lower bound from 7023242018695589888 to 7023242017999933440. Current Snapshot: MvccSnapshot[applied={T|T < 7023242018695589888}]
I20240502 14:04:56.011252 30673 mvcc.cc:205] Tried to move back new op lower bound from 7023242018695589888 to 7023242017999933440. Current Snapshot: MvccSnapshot[applied={T|T < 7023242018695589888}]
I20240502 14:04:56.021510 30653 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.022295 30653 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 2febb32cb57e4de6836a2050e0cd3046: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:56.026230 30676 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.027066 30676 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:56.038317 30676 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.039203 30676 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P fdd4b183fbd747dd89385c76fb97f2f2: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:04:56.049274 30680 mvcc.cc:205] Tried to move back new op lower bound from 7023242018695589888 to 7023242017999933440. Current Snapshot: MvccSnapshot[applied={T|T < 7023242018695589888}]
I20240502 14:04:56.078253 30652 tablet_replica.cc:439] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } }
I20240502 14:04:56.079264 30652 tablet_replica.cc:442] T c8ea78345ebe4e1aaba5e6e60f6ba85f P 25cad6faa1e745fbba1db358f5313861: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:04:56.090183 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:56.064814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.27.174.67:41667)
0502 14:04:56.065597 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:56.065627 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:56.076733 (+ 11106us) server_negotiation.cc:407] Connection header received
0502 14:04:56.080642 (+ 3909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:56.080672 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:56.080977 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:56.081103 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:56.082792 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.084079 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:56.086297 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.086844 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:56.087552 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:56.087863 (+ 311us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:56.087984 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:56.088416 (+ 432us) server_negotiation.cc:299] Negotiation successful
0502 14:04:56.088604 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20240502 14:04:56.148703 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:56.129257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:39279 (local address 127.27.174.67:41667)
0502 14:04:56.130180 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:56.130232 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:56.138585 (+ 8353us) server_negotiation.cc:407] Connection header received
0502 14:04:56.139634 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:56.139673 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:56.139943 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:56.140057 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:56.141638 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.142910 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:56.145108 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.145611 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:56.146346 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:56.146654 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:56.146768 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:56.147131 (+ 363us) server_negotiation.cc:299] Negotiation successful
0502 14:04:56.147306 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20240502 14:04:56.151341 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:56.127687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41667 (local address 127.27.174.67:39279)
0502 14:04:56.138384 (+ 10697us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:56.138442 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:56.139374 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:56.140394 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:56.140420 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:56.140466 (+ 46us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:56.141395 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:56.141438 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.143338 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:56.143374 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:56.144855 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:56.144907 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:56.145906 (+ 999us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:56.146025 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:56.146896 (+ 871us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:56.146931 (+ 35us) client_negotiation.cc:769] Sending connection context
0502 14:04:56.149335 (+ 2404us) client_negotiation.cc:240] Negotiation successful
0502 14:04:56.149538 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9979,"thread_start_us":10846,"threads_started":1}
I20240502 14:04:56.379849 30133 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:56.380503 30133 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:56.396018 30133 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Sending DeleteTablet for 3 replicas of tablet 2573ea7ce02a4f4f8459334f188d9bdb
I20240502 14:04:56.398588 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:56 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:41620
I20240502 14:04:56.399014 30562 tablet_service.cc:1514] Processing DeleteTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:56 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57814
I20240502 14:04:56.401468 30278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2573ea7ce02a4f4f8459334f188d9bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:56 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47458
I20240502 14:04:56.403542 30133 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:56.405437 30133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:56.414923 30705 tablet_replica.cc:330] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: stopping tablet replica
I20240502 14:04:56.415933 30705 raft_consensus.cc:2229] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:56.416602 30705 raft_consensus.cc:2258] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:56.420596 30705 ts_tablet_manager.cc:1875] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:56.453703 30705 ts_tablet_manager.cc:1888] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:56.455061 30705 log.cc:1196] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/2573ea7ce02a4f4f8459334f188d9bdb
I20240502 14:04:56.458204 30419 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae (DEFAULT_TABLE table=test-table [id=eb46062c4982476189c067f265d48f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:56.459467 30419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc1b5dd047346a2bf5c2127891d84ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:56.459849 30561 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae (DEFAULT_TABLE table=test-table [id=eb46062c4982476189c067f265d48f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:56.455498 30278 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae (DEFAULT_TABLE table=test-table [id=eb46062c4982476189c067f265d48f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:56.461104 30561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc1b5dd047346a2bf5c2127891d84ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:56.461432 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc1b5dd047346a2bf5c2127891d84ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:56.463685 30705 ts_tablet_manager.cc:1909] T 2573ea7ce02a4f4f8459334f188d9bdb P 2febb32cb57e4de6836a2050e0cd3046: Deleting consensus metadata
I20240502 14:04:56.469323 30118 catalog_manager.cc:4830] TS 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865): tablet 2573ea7ce02a4f4f8459334f188d9bdb (table test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]) successfully deleted
I20240502 14:04:56.475409 30704 tablet_replica.cc:330] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: stopping tablet replica
I20240502 14:04:56.481123 30706 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Bootstrap starting.
I20240502 14:04:56.483170 30704 raft_consensus.cc:2229] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:56.487732 30704 raft_consensus.cc:2258] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:56.489544 30707 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Bootstrap starting.
I20240502 14:04:56.491690 30704 ts_tablet_manager.cc:1875] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:56.491681 30703 tablet_replica.cc:330] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: stopping tablet replica
I20240502 14:04:56.492761 30703 raft_consensus.cc:2229] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:56.495947 30703 raft_consensus.cc:2258] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:56.496466 30707 tablet_bootstrap.cc:654] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:56.499159 30708 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Bootstrap starting.
I20240502 14:04:56.504870 30708 tablet_bootstrap.cc:654] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:56.505407 30703 ts_tablet_manager.cc:1875] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:56.507064 30706 tablet_bootstrap.cc:654] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:56.516906 30704 ts_tablet_manager.cc:1888] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:56.517506 30704 log.cc:1196] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/2573ea7ce02a4f4f8459334f188d9bdb
I20240502 14:04:56.518585 30704 ts_tablet_manager.cc:1909] T 2573ea7ce02a4f4f8459334f188d9bdb P fdd4b183fbd747dd89385c76fb97f2f2: Deleting consensus metadata
I20240502 14:04:56.520700 30708 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: No bootstrap required, opened a new log
I20240502 14:04:56.521167 30708 ts_tablet_manager.cc:1367] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240502 14:04:56.521415 30707 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: No bootstrap required, opened a new log
I20240502 14:04:56.521643 30118 catalog_manager.cc:4830] TS fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667): tablet 2573ea7ce02a4f4f8459334f188d9bdb (table test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]) successfully deleted
I20240502 14:04:56.521838 30707 ts_tablet_manager.cc:1367] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20240502 14:04:56.524284 30707 raft_consensus.cc:348] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.524463 30708 raft_consensus.cc:348] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.525045 30707 raft_consensus.cc:374] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.525187 30708 raft_consensus.cc:374] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.525395 30707 raft_consensus.cc:729] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febb32cb57e4de6836a2050e0cd3046, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.525480 30708 raft_consensus.cc:729] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.526199 30707 consensus_queue.cc:260] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.527557 30706 tablet_bootstrap.cc:492] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: No bootstrap required, opened a new log
I20240502 14:04:56.527949 30706 ts_tablet_manager.cc:1367] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.008s
I20240502 14:04:56.534308 30708 consensus_queue.cc:260] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.534901 30703 ts_tablet_manager.cc:1888] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:56.535336 30703 log.cc:1196] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/2573ea7ce02a4f4f8459334f188d9bdb
I20240502 14:04:56.530454 30706 raft_consensus.cc:348] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.535830 30706 raft_consensus.cc:374] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:56.536118 30707 ts_tablet_manager.cc:1398] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Time spent starting tablet: real 0.014s user 0.003s sys 0.008s
I20240502 14:04:56.536109 30706 raft_consensus.cc:729] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Initialized, Role: FOLLOWER
I20240502 14:04:56.536435 30703 ts_tablet_manager.cc:1909] T 2573ea7ce02a4f4f8459334f188d9bdb P 25cad6faa1e745fbba1db358f5313861: Deleting consensus metadata
I20240502 14:04:56.536733 30706 consensus_queue.cc:260] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.541980 30708 ts_tablet_manager.cc:1398] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240502 14:04:56.544226 30706 ts_tablet_manager.cc:1398] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20240502 14:04:56.552153 30121 catalog_manager.cc:4830] TS 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735): tablet 2573ea7ce02a4f4f8459334f188d9bdb (table test-table [id=f045bbf5c34a430586a2cdac55fb7e2b]) successfully deleted
I20240502 14:04:56.596598 30676 raft_consensus.cc:482] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:56.597087 30676 raft_consensus.cc:504] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.598973 30676 leader_election.cc:288] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:56.602433 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc1b5dd047346a2bf5c2127891d84ae" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:56.603082 30298 raft_consensus.cc:2454] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:56.604878 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc1b5dd047346a2bf5c2127891d84ae" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861" is_pre_election: true
I20240502 14:04:56.605479 30439 raft_consensus.cc:2454] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:56.606828 30507 leader_election.cc:302] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:56.610764 30676 raft_consensus.cc:2781] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:56.611610 30676 raft_consensus.cc:482] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:56.612038 30676 raft_consensus.cc:3037] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.619333 30676 raft_consensus.cc:504] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.622300 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc1b5dd047346a2bf5c2127891d84ae" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046"
I20240502 14:04:56.622889 30298 raft_consensus.cc:3037] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.630322 30298 raft_consensus.cc:2454] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:56.631649 30507 leader_election.cc:302] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:56.632586 30651 raft_consensus.cc:2781] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:56.633000 30651 raft_consensus.cc:686] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Becoming Leader. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Running, Role: LEADER
I20240502 14:04:56.633797 30651 consensus_queue.cc:237] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:56.645871 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc1b5dd047346a2bf5c2127891d84ae" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861"
I20240502 14:04:56.646493 30439 raft_consensus.cc:3037] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:56.653856 30439 raft_consensus.cc:2454] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:56.655663 30676 leader_election.cc:288] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Requested vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:56.666801 30134 catalog_manager.cc:5484] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:56.760588 30439 raft_consensus.cc:1261] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:56.761055 30298 raft_consensus.cc:1261] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:56.762411 30676 consensus_queue.cc:1035] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:56.763120 30651 consensus_queue.cc:1035] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:04:56.876869 30672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
W20240502 14:04:58.664417 30339 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcaa0 after lost signal to thread 30206
W20240502 14:04:58.665812 30339 debug-util.cc:398] Leaking SignalData structure 0x7b08000de580 after lost signal to thread 30342
I20240502 14:04:58.764896 30132 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:58.765601 30132 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:58.777467 30132 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Sending DeleteTablet for 3 replicas of tablet 5bc1b5dd047346a2bf5c2127891d84ae
I20240502 14:04:58.779796 30278 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:58 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47458
I20240502 14:04:58.781246 30741 tablet_replica.cc:330] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: stopping tablet replica
I20240502 14:04:58.782047 30741 raft_consensus.cc:2229] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:58.782346 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:58 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:41620
I20240502 14:04:58.782663 30741 raft_consensus.cc:2258] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:58.784266 30561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc1b5dd047346a2bf5c2127891d84ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:58 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57814
I20240502 14:04:58.785568 30741 ts_tablet_manager.cc:1875] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:58.804245 30741 ts_tablet_manager.cc:1888] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240502 14:04:58.804740 30741 log.cc:1196] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/5bc1b5dd047346a2bf5c2127891d84ae
I20240502 14:04:58.805636 30741 ts_tablet_manager.cc:1909] T 5bc1b5dd047346a2bf5c2127891d84ae P 2febb32cb57e4de6836a2050e0cd3046: Deleting consensus metadata
I20240502 14:04:58.808693 30118 catalog_manager.cc:4830] TS 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865): tablet 5bc1b5dd047346a2bf5c2127891d84ae (table test-table [id=eb46062c4982476189c067f265d48f79]) successfully deleted
I20240502 14:04:58.813099 30743 tablet_replica.cc:330] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: stopping tablet replica
I20240502 14:04:58.814193 30742 tablet_replica.cc:330] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: stopping tablet replica
I20240502 14:04:58.815025 30743 raft_consensus.cc:2229] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:58.815850 30742 raft_consensus.cc:2229] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:58.816848 30743 raft_consensus.cc:2258] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:58.817385 30742 raft_consensus.cc:2258] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:58.820763 30742 ts_tablet_manager.cc:1875] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:58.825016 30743 ts_tablet_manager.cc:1875] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:58.846854 30743 ts_tablet_manager.cc:1888] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240502 14:04:58.847324 30743 log.cc:1196] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/5bc1b5dd047346a2bf5c2127891d84ae
I20240502 14:04:58.848101 30743 ts_tablet_manager.cc:1909] T 5bc1b5dd047346a2bf5c2127891d84ae P fdd4b183fbd747dd89385c76fb97f2f2: Deleting consensus metadata
I20240502 14:04:58.850798 30118 catalog_manager.cc:4830] TS fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667): tablet 5bc1b5dd047346a2bf5c2127891d84ae (table test-table [id=eb46062c4982476189c067f265d48f79]) successfully deleted
I20240502 14:04:58.851146 30742 ts_tablet_manager.cc:1888] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240502 14:04:58.851604 30742 log.cc:1196] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/5bc1b5dd047346a2bf5c2127891d84ae
I20240502 14:04:58.852392 30742 ts_tablet_manager.cc:1909] T 5bc1b5dd047346a2bf5c2127891d84ae P 25cad6faa1e745fbba1db358f5313861: Deleting consensus metadata
I20240502 14:04:58.855070 30121 catalog_manager.cc:4830] TS 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735): tablet 5bc1b5dd047346a2bf5c2127891d84ae (table test-table [id=eb46062c4982476189c067f265d48f79]) successfully deleted
May 02 14:04:58 dist-test-slave-dlb8 krb5kdc[30086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 02 14:04:58 dist-test-slave-dlb8 krb5kdc[30086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:04:58.912837 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:58.876835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.27.174.126:36253)
0502 14:04:58.893043 (+ 16208us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:58.893065 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:58.893108 (+ 43us) server_negotiation.cc:407] Connection header received
0502 14:04:58.893262 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:58.893277 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:58.893465 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:58.893633 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:04:58.896400 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:58.897608 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:58.899054 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:58.899645 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:58.903699 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:04:58.903781 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:04:58.903799 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:04:58.903884 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:04:58.907653 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:58.908604 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:58.908624 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:58.908648 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:58.908808 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:04:58.909319 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:04:58.909338 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:04:58.909352 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:04:58.909733 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:04:58.910145 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:58.910710 (+ 565us) server_negotiation.cc:299] Negotiation successful
0502 14:04:58.911183 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15735,"thread_start_us":17007,"threads_started":1}
I20240502 14:04:58.918536 30132 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:58.920125 30132 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:58.937711 30278 tablet_service.cc:1467] Processing CreateTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (DEFAULT_TABLE table=test-table [id=d19f46c7a2684fd0abae478d10ea95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:58.938323 30561 tablet_service.cc:1467] Processing CreateTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (DEFAULT_TABLE table=test-table [id=d19f46c7a2684fd0abae478d10ea95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:58.938916 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d123f05564b4ed7b4cdd4900dd94aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:58.939334 30561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d123f05564b4ed7b4cdd4900dd94aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:58.948204 30420 tablet_service.cc:1467] Processing CreateTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (DEFAULT_TABLE table=test-table [id=d19f46c7a2684fd0abae478d10ea95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:58.949369 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d123f05564b4ed7b4cdd4900dd94aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:58.957163 30753 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Bootstrap starting.
I20240502 14:04:58.962080 30752 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Bootstrap starting.
I20240502 14:04:58.968559 30752 tablet_bootstrap.cc:654] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:58.969774 30753 tablet_bootstrap.cc:654] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:58.980979 30754 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Bootstrap starting.
I20240502 14:04:58.984807 30753 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: No bootstrap required, opened a new log
I20240502 14:04:58.985306 30753 ts_tablet_manager.cc:1367] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.007s
I20240502 14:04:58.987643 30753 raft_consensus.cc:348] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:58.988435 30753 raft_consensus.cc:374] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:58.988749 30753 raft_consensus.cc:729] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Initialized, Role: FOLLOWER
I20240502 14:04:58.989423 30753 consensus_queue.cc:260] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:58.993182 30752 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: No bootstrap required, opened a new log
I20240502 14:04:58.993408 30753 ts_tablet_manager.cc:1398] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:04:58.993634 30752 ts_tablet_manager.cc:1367] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.019s
I20240502 14:04:58.996665 30754 tablet_bootstrap.cc:654] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:58.996378 30752 raft_consensus.cc:348] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:58.997241 30752 raft_consensus.cc:374] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:58.997529 30752 raft_consensus.cc:729] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febb32cb57e4de6836a2050e0cd3046, State: Initialized, Role: FOLLOWER
I20240502 14:04:58.998052 30752 consensus_queue.cc:260] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.006127 30754 tablet_bootstrap.cc:492] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: No bootstrap required, opened a new log
I20240502 14:04:59.006565 30754 ts_tablet_manager.cc:1367] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.000s
I20240502 14:04:59.007701 30752 ts_tablet_manager.cc:1398] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
I20240502 14:04:59.008859 30754 raft_consensus.cc:348] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.009584 30754 raft_consensus.cc:374] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:59.009892 30754 raft_consensus.cc:729] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Initialized, Role: FOLLOWER
I20240502 14:04:59.010524 30754 consensus_queue.cc:260] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.015040 30754 ts_tablet_manager.cc:1398] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:04:59.122535 30736 raft_consensus.cc:482] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:59.123014 30736 raft_consensus.cc:504] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.124804 30736 leader_election.cc:288] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:59.125828 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d123f05564b4ed7b4cdd4900dd94aa9" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:59.126268 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d123f05564b4ed7b4cdd4900dd94aa9" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861" is_pre_election: true
I20240502 14:04:59.126441 30298 raft_consensus.cc:2454] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:59.126808 30439 raft_consensus.cc:2454] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 0.
I20240502 14:04:59.127550 30507 leader_election.cc:302] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2febb32cb57e4de6836a2050e0cd3046, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:59.128247 30736 raft_consensus.cc:2781] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:59.128533 30736 raft_consensus.cc:482] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:59.128976 30736 raft_consensus.cc:3037] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.135720 30736 raft_consensus.cc:504] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.138195 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d123f05564b4ed7b4cdd4900dd94aa9" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046"
I20240502 14:04:59.138329 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d123f05564b4ed7b4cdd4900dd94aa9" candidate_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cad6faa1e745fbba1db358f5313861"
I20240502 14:04:59.138738 30298 raft_consensus.cc:3037] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.138852 30439 raft_consensus.cc:3037] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.145373 30439 raft_consensus.cc:2454] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:59.145545 30298 raft_consensus.cc:2454] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd4b183fbd747dd89385c76fb97f2f2 in term 1.
I20240502 14:04:59.146461 30510 leader_election.cc:302] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:59.147162 30736 leader_election.cc:288] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [CANDIDATE]: Term 1 election: Requested vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735)
I20240502 14:04:59.147320 30723 raft_consensus.cc:2781] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:59.147783 30723 raft_consensus.cc:686] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Becoming Leader. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Running, Role: LEADER
I20240502 14:04:59.148524 30723 consensus_queue.cc:237] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } }
I20240502 14:04:59.158198 30134 catalog_manager.cc:5484] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:59.206331 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:59.197891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.27.174.67:41667)
0502 14:04:59.199467 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:59.199489 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:59.199534 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:04:59.199678 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:59.199705 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:59.199998 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:59.200141 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:59.200834 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.202449 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:59.203601 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.204164 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:59.204352 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:59.204621 (+ 269us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:59.204776 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:59.205021 (+ 245us) server_negotiation.cc:299] Negotiation successful
0502 14:04:59.205193 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":720,"threads_started":1}
I20240502 14:04:59.227005 30439 raft_consensus.cc:1261] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:59.227226 30297 raft_consensus.cc:1261] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Refusing update from remote peer fdd4b183fbd747dd89385c76fb97f2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:59.228228 30723 consensus_queue.cc:1035] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:59.229054 30736 consensus_queue.cc:1035] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:59.387967 30133 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:59.388500 30133 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:04:59.395720 30133 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Sending DeleteTablet for 3 replicas of tablet 0d123f05564b4ed7b4cdd4900dd94aa9
I20240502 14:04:59.397419 30278 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:59 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47458
I20240502 14:04:59.397766 30561 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:59 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57814
I20240502 14:04:59.398026 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d123f05564b4ed7b4cdd4900dd94aa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:04:59 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:41620
I20240502 14:04:59.403725 30133 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:59.406692 30133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:59.408860 30768 tablet_replica.cc:330] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: stopping tablet replica
I20240502 14:04:59.409853 30768 raft_consensus.cc:2229] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:59.410938 30768 raft_consensus.cc:2258] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:59.411501 30767 tablet_replica.cc:330] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: stopping tablet replica
I20240502 14:04:59.412465 30767 raft_consensus.cc:2229] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:59.413030 30767 raft_consensus.cc:2258] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:59.416189 30768 ts_tablet_manager.cc:1875] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:59.416548 30767 ts_tablet_manager.cc:1875] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:59.431000 30768 ts_tablet_manager.cc:1888] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:59.430958 30767 ts_tablet_manager.cc:1888] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:59.431340 30768 log.cc:1196] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/0d123f05564b4ed7b4cdd4900dd94aa9
I20240502 14:04:59.431504 30767 log.cc:1196] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/0d123f05564b4ed7b4cdd4900dd94aa9
I20240502 14:04:59.432024 30768 ts_tablet_manager.cc:1909] T 0d123f05564b4ed7b4cdd4900dd94aa9 P fdd4b183fbd747dd89385c76fb97f2f2: Deleting consensus metadata
I20240502 14:04:59.432341 30767 ts_tablet_manager.cc:1909] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 2febb32cb57e4de6836a2050e0cd3046: Deleting consensus metadata
I20240502 14:04:59.434393 30118 catalog_manager.cc:4830] TS fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667): tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (table test-table [id=d19f46c7a2684fd0abae478d10ea95f0]) successfully deleted
I20240502 14:04:59.434465 30278 tablet_service.cc:1467] Processing CreateTablet for tablet 708e448514124be3a77779830196b66b (DEFAULT_TABLE table=test-table [id=9cd217dac3bf4dce853aab8fdf201f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:59.435001 30118 catalog_manager.cc:4830] TS 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865): tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (table test-table [id=d19f46c7a2684fd0abae478d10ea95f0]) successfully deleted
I20240502 14:04:59.435446 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708e448514124be3a77779830196b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:59.438390 30561 tablet_service.cc:1467] Processing CreateTablet for tablet 708e448514124be3a77779830196b66b (DEFAULT_TABLE table=test-table [id=9cd217dac3bf4dce853aab8fdf201f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:59.439592 30561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708e448514124be3a77779830196b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:59.440081 30769 tablet_replica.cc:330] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: stopping tablet replica
I20240502 14:04:59.440757 30769 raft_consensus.cc:2229] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:59.441171 30420 tablet_service.cc:1467] Processing CreateTablet for tablet 708e448514124be3a77779830196b66b (DEFAULT_TABLE table=test-table [id=9cd217dac3bf4dce853aab8fdf201f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:04:59.441376 30769 raft_consensus.cc:2258] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:59.443367 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708e448514124be3a77779830196b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:59.444566 30769 ts_tablet_manager.cc:1875] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:04:59.456013 30753 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Bootstrap starting.
I20240502 14:04:59.462639 30753 tablet_bootstrap.cc:654] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:59.467233 30752 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: Bootstrap starting.
I20240502 14:04:59.472108 30769 ts_tablet_manager.cc:1888] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:04:59.472476 30769 log.cc:1196] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/0d123f05564b4ed7b4cdd4900dd94aa9
I20240502 14:04:59.473100 30752 tablet_bootstrap.cc:654] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:59.473152 30769 ts_tablet_manager.cc:1909] T 0d123f05564b4ed7b4cdd4900dd94aa9 P 25cad6faa1e745fbba1db358f5313861: Deleting consensus metadata
I20240502 14:04:59.474311 30753 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: No bootstrap required, opened a new log
I20240502 14:04:59.474686 30753 ts_tablet_manager.cc:1367] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240502 14:04:59.476828 30753 raft_consensus.cc:348] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.477475 30753 raft_consensus.cc:374] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:59.477720 30753 raft_consensus.cc:729] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd4b183fbd747dd89385c76fb97f2f2, State: Initialized, Role: FOLLOWER
I20240502 14:04:59.478462 30753 consensus_queue.cc:260] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.478828 30121 catalog_manager.cc:4830] TS 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735): tablet 0d123f05564b4ed7b4cdd4900dd94aa9 (table test-table [id=d19f46c7a2684fd0abae478d10ea95f0]) successfully deleted
I20240502 14:04:59.485049 30754 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Bootstrap starting.
I20240502 14:04:59.487651 30753 ts_tablet_manager.cc:1398] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:04:59.490429 30754 tablet_bootstrap.cc:654] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:59.492197 30752 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: No bootstrap required, opened a new log
I20240502 14:04:59.492579 30752 ts_tablet_manager.cc:1367] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.013s
I20240502 14:04:59.494710 30752 raft_consensus.cc:348] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.495414 30752 raft_consensus.cc:374] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:59.495625 30752 raft_consensus.cc:729] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febb32cb57e4de6836a2050e0cd3046, State: Initialized, Role: FOLLOWER
I20240502 14:04:59.496109 30752 consensus_queue.cc:260] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.496619 30754 tablet_bootstrap.cc:492] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: No bootstrap required, opened a new log
I20240502 14:04:59.498471 30752 ts_tablet_manager.cc:1398] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 14:04:59.499348 30754 ts_tablet_manager.cc:1367] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240502 14:04:59.500959 30754 raft_consensus.cc:348] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.501576 30754 raft_consensus.cc:374] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:59.501847 30754 raft_consensus.cc:729] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Initialized, Role: FOLLOWER
I20240502 14:04:59.502444 30754 consensus_queue.cc:260] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.504187 30754 ts_tablet_manager.cc:1398] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:04:59.781530 30759 raft_consensus.cc:482] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:59.782075 30759 raft_consensus.cc:504] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.783859 30759 leader_election.cc:288] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667)
I20240502 14:04:59.784938 30582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708e448514124be3a77779830196b66b" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" is_pre_election: true
I20240502 14:04:59.785424 30582 raft_consensus.cc:2454] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25cad6faa1e745fbba1db358f5313861 in term 0.
I20240502 14:04:59.785300 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708e448514124be3a77779830196b66b" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046" is_pre_election: true
I20240502 14:04:59.786303 30365 leader_election.cc:302] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861, fdd4b183fbd747dd89385c76fb97f2f2; no voters:
I20240502 14:04:59.786314 30298 raft_consensus.cc:2454] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25cad6faa1e745fbba1db358f5313861 in term 0.
I20240502 14:04:59.786850 30759 raft_consensus.cc:2781] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:59.787106 30759 raft_consensus.cc:482] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:59.787412 30759 raft_consensus.cc:3037] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.791966 30759 raft_consensus.cc:504] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.793296 30759 leader_election.cc:288] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 election: Requested vote from peers 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865), fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667)
I20240502 14:04:59.794076 30298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708e448514124be3a77779830196b66b" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febb32cb57e4de6836a2050e0cd3046"
I20240502 14:04:59.794445 30298 raft_consensus.cc:3037] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.794351 30582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708e448514124be3a77779830196b66b" candidate_uuid: "25cad6faa1e745fbba1db358f5313861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd4b183fbd747dd89385c76fb97f2f2"
I20240502 14:04:59.794857 30582 raft_consensus.cc:3037] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:59.800714 30298 raft_consensus.cc:2454] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25cad6faa1e745fbba1db358f5313861 in term 1.
I20240502 14:04:59.801404 30582 raft_consensus.cc:2454] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25cad6faa1e745fbba1db358f5313861 in term 1.
I20240502 14:04:59.801530 30365 leader_election.cc:302] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cad6faa1e745fbba1db358f5313861, 2febb32cb57e4de6836a2050e0cd3046; no voters:
I20240502 14:04:59.802618 30759 raft_consensus.cc:2781] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:59.803594 30759 raft_consensus.cc:686] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 1 LEADER]: Becoming Leader. State: Replica: 25cad6faa1e745fbba1db358f5313861, State: Running, Role: LEADER
I20240502 14:04:59.804373 30759 consensus_queue.cc:237] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } }
I20240502 14:04:59.813068 30132 catalog_manager.cc:5484] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25cad6faa1e745fbba1db358f5313861 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "25cad6faa1e745fbba1db358f5313861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25cad6faa1e745fbba1db358f5313861" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 42735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:59.901885 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:59.882409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.27.174.66:42735)
0502 14:04:59.891046 (+ 8637us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:59.891288 (+ 242us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:59.892993 (+ 1705us) server_negotiation.cc:407] Connection header received
0502 14:04:59.893441 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:59.893468 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:59.893753 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:59.893927 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:59.895022 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.896606 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:59.898315 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.898799 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:59.899018 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:59.899772 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:59.899956 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:59.900245 (+ 289us) server_negotiation.cc:299] Negotiation successful
0502 14:04:59.900405 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7251,"thread_start_us":6921,"threads_started":1}
I20240502 14:04:59.941073 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:59.920559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39659 (local address 127.27.174.67:41667)
0502 14:04:59.931194 (+ 10635us) server_negotiation.cc:206] Beginning negotiation
0502 14:04:59.931220 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:04:59.933047 (+ 1827us) server_negotiation.cc:407] Connection header received
0502 14:04:59.933229 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:04:59.933255 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:04:59.933540 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:04:59.933684 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:04:59.935916 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.936797 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:04:59.938152 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.938643 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:59.938758 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:04:59.938972 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:04:59.939098 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0502 14:04:59.939366 (+ 268us) server_negotiation.cc:299] Negotiation successful
0502 14:04:59.939526 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10037,"thread_start_us":9852,"threads_started":1}
I20240502 14:04:59.941490 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:04:59.920488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41667 (local address 127.27.174.66:39659)
0502 14:04:59.932516 (+ 12028us) negotiation.cc:107] Waiting for socket to connect
0502 14:04:59.932563 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0502 14:04:59.932853 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:04:59.934023 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:04:59.934073 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:04:59.934125 (+ 52us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:04:59.935600 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:59.935645 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.937023 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:04:59.937045 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:04:59.937927 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:04:59.937958 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:04:59.938276 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:04:59.938403 (+ 127us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:04:59.939153 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:04:59.939181 (+ 28us) client_negotiation.cc:769] Sending connection context
0502 14:04:59.939289 (+ 108us) client_negotiation.cc:240] Negotiation successful
0502 14:04:59.939512 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11394,"thread_start_us":12229,"threads_started":1}
I20240502 14:04:59.959056 30298 raft_consensus.cc:1261] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Refusing update from remote peer 25cad6faa1e745fbba1db358f5313861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:59.959273 30582 raft_consensus.cc:1261] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Refusing update from remote peer 25cad6faa1e745fbba1db358f5313861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:59.960579 30774 consensus_queue.cc:1035] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2febb32cb57e4de6836a2050e0cd3046" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 33865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:59.961620 30759 consensus_queue.cc:1035] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd4b183fbd747dd89385c76fb97f2f2" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:00.032346 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:00.021805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33845 (local address 127.27.174.66:42735)
0502 14:05:00.022537 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:00.022564 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:00.024468 (+ 1904us) server_negotiation.cc:407] Connection header received
0502 14:05:00.024883 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:00.024905 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:00.025168 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:00.025320 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:05:00.026706 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:00.027827 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:00.029779 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:00.030376 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:00.030539 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:05:00.030787 (+ 248us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:05:00.030939 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:00.031280 (+ 341us) server_negotiation.cc:299] Negotiation successful
0502 14:05:00.031390 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20240502 14:05:00.032877 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:00.021652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:42735 (local address 127.27.174.67:33845)
0502 14:05:00.024276 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:00.024340 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:00.024636 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:00.025398 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:00.025430 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:00.025484 (+ 54us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:05:00.026422 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:00.026481 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:00.028088 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:00.028119 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:00.029491 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:00.029557 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:00.029910 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:00.030140 (+ 230us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:05:00.031051 (+ 911us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:05:00.031086 (+ 35us) client_negotiation.cc:769] Sending connection context
0502 14:05:00.031209 (+ 123us) client_negotiation.cc:240] Negotiation successful
0502 14:05:00.031414 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2073,"thread_start_us":1760,"threads_started":1}
W20240502 14:05:00.034834 30724 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240502 14:05:01.829092 30133 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:01.829670 30133 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:01.840510 30133 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 06d26386e9524101b619fa9994abae88: Sending DeleteTablet for 3 replicas of tablet 708e448514124be3a77779830196b66b
I20240502 14:05:01.842393 30278 tablet_service.cc:1514] Processing DeleteTablet for tablet 708e448514124be3a77779830196b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47458
I20240502 14:05:01.843859 30561 tablet_service.cc:1514] Processing DeleteTablet for tablet 708e448514124be3a77779830196b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:57814
I20240502 14:05:01.845433 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet 708e448514124be3a77779830196b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:01 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:41620
I20240502 14:05:01.854959 30819 tablet_replica.cc:330] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: stopping tablet replica
I20240502 14:05:01.855657 30819 raft_consensus.cc:2229] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:01.856129 30819 raft_consensus.cc:2258] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:01.859128 30819 ts_tablet_manager.cc:1875] T 708e448514124be3a77779830196b66b P 2febb32cb57e4de6836a2050e0cd3046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:01.860690 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30205
I20240502 14:05:01.867760 30820 tablet_replica.cc:330] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: stopping tablet replica
I20240502 14:05:01.868216 30821 tablet_replica.cc:330] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: stopping tablet replica
I20240502 14:05:01.868625 30820 raft_consensus.cc:2229] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:01.869064 30821 raft_consensus.cc:2229] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:01.869271 30820 raft_consensus.cc:2258] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:01.870303 30821 raft_consensus.cc:2258] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:01.872774 30820 ts_tablet_manager.cc:1875] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:01.874215 30821 ts_tablet_manager.cc:1875] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:01.891463 30821 ts_tablet_manager.cc:1888] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240502 14:05:01.892040 30821 log.cc:1196] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/708e448514124be3a77779830196b66b
I20240502 14:05:01.892792 30820 ts_tablet_manager.cc:1888] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240502 14:05:01.893347 30820 log.cc:1196] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/708e448514124be3a77779830196b66b
I20240502 14:05:01.893308 30821 ts_tablet_manager.cc:1909] T 708e448514124be3a77779830196b66b P 25cad6faa1e745fbba1db358f5313861: Deleting consensus metadata
I20240502 14:05:01.894155 30820 ts_tablet_manager.cc:1909] T 708e448514124be3a77779830196b66b P fdd4b183fbd747dd89385c76fb97f2f2: Deleting consensus metadata
I20240502 14:05:01.896402 30121 catalog_manager.cc:4830] TS 25cad6faa1e745fbba1db358f5313861 (127.27.174.66:42735): tablet 708e448514124be3a77779830196b66b (table test-table [id=9cd217dac3bf4dce853aab8fdf201f9a]) successfully deleted
I20240502 14:05:01.896888 30118 catalog_manager.cc:4830] TS fdd4b183fbd747dd89385c76fb97f2f2 (127.27.174.67:41667): tablet 708e448514124be3a77779830196b66b (table test-table [id=9cd217dac3bf4dce853aab8fdf201f9a]) successfully deleted
I20240502 14:05:01.901396 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30347
W20240502 14:05:01.901664 30118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:33865 (error 108)
I20240502 14:05:01.909286 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:01.906418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.0.0.1:51426)
0502 14:05:01.908159 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:01.908417 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":660,"threads_started":1}
W20240502 14:05:01.909785 30118 catalog_manager.cc:4573] TS 2febb32cb57e4de6836a2050e0cd3046 (127.27.174.65:33865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 708e448514124be3a77779830196b66b: Network error: Client connection negotiation failed: client connection to 127.27.174.65:33865: connect: Connection refused (error 111)
I20240502 14:05:01.937136 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30489
I20240502 14:05:01.965224 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:01.963495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.0.0.1:51442)
0502 14:05:01.964364 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:01.964612 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20240502 14:05:01.968261 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:01.966590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:33865 (local address 127.0.0.1:51456)
0502 14:05:01.967480 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:01.967697 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20240502 14:05:01.980389 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30100
2024-05-02T14:05:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (15645 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:02 dist-test-slave-dlb8 krb5kdc[30826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:02 dist-test-slave-dlb8 krb5kdc[30826](info): set up 2 sockets
May 02 14:05:02 dist-test-slave-dlb8 krb5kdc[30826](info): commencing operation
krb5kdc: starting...
W20240502 14:05:02.987422 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.009s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:03 dist-test-slave-dlb8 krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:03.192986 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34649
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:03.475885 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:03.476456 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:03.477008 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:03.505934 30839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:03.506305 30839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:03.506678 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:03.507030 30839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:03.507347 30839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:03.542790 30839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:03.543871 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:03.544816 30839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:03.559484 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:03.560298 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:03.561256 30839 server_base.cc:1034] running on GCE node
W20240502 14:05:03.561035 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:04.591470 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:04.594594 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:04.596011 30839 hybrid_clock.cc:648] HybridClock initialized: now 1714658704595987 us; error 47 us; skew 500 ppm
I20240502 14:05:04.596827 30839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:04 dist-test-slave-dlb8 krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658704, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:04.602438 30839 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:05:04.608016 30839 webserver.cc:460] Webserver started at http://127.27.174.126:43893/ using document root <none> and password file <none>
I20240502 14:05:04.609122 30839 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:04.609367 30839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:04.609756 30839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:04.614738 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "293fb3dde84247f88a9c33df60fb0130"
format_stamp: "Formatted at 2024-05-02 14:05:04 on dist-test-slave-dlb8"
server_key: "ca2eb27182e3e81a55c09209c51e7be2"
server_key_iv: "f6b23571d5547eb3f72a6149715e3ef8"
server_key_version: "encryptionkey@0"
I20240502 14:05:04.615828 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "293fb3dde84247f88a9c33df60fb0130"
format_stamp: "Formatted at 2024-05-02 14:05:04 on dist-test-slave-dlb8"
server_key: "ca2eb27182e3e81a55c09209c51e7be2"
server_key_iv: "f6b23571d5547eb3f72a6149715e3ef8"
server_key_version: "encryptionkey@0"
I20240502 14:05:04.622753 30839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 14:05:04.628031 30856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:04.629032 30839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:05:04.629369 30839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "293fb3dde84247f88a9c33df60fb0130"
format_stamp: "Formatted at 2024-05-02 14:05:04 on dist-test-slave-dlb8"
server_key: "ca2eb27182e3e81a55c09209c51e7be2"
server_key_iv: "f6b23571d5547eb3f72a6149715e3ef8"
server_key_version: "encryptionkey@0"
I20240502 14:05:04.629717 30839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:04.689733 30839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:04.695185 30839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:04.695596 30839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:04.761406 30839 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34649
I20240502 14:05:04.761480 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34649 every 8 connection(s)
I20240502 14:05:04.764129 30839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:04.767268 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30839
I20240502 14:05:04.767742 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:04.769203 28345 external_mini_cluster.cc:1428] Setting key e004985ba8c9c2307feab823ef3451c8
I20240502 14:05:04.770184 30908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:05:04 dist-test-slave-dlb8 krb5kdc[30826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:04.795607 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130: Bootstrap starting.
I20240502 14:05:04.801307 30908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:04.802816 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:04.772053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.27.174.126:34649)
0502 14:05:04.774582 (+ 2529us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:04.774607 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:04.774653 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:05:04.779505 (+ 4852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:04.779546 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:04.779893 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:04.780228 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:04.783620 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:04.784981 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:04.786724 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:04.787098 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:04.792859 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:04.792952 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:04.792979 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:04.793076 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:04.796788 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:04.797962 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:04.797981 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:04.798009 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:04.798199 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:04.798765 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:04.798782 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:04.798795 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:04.799339 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:04.799692 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:04.800391 (+ 699us) server_negotiation.cc:299] Negotiation successful
0502 14:05:04.800814 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2054,"thread_start_us":1620,"threads_started":1}
I20240502 14:05:04.803556 30908 log.cc:824] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:04.808282 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130: No bootstrap required, opened a new log
I20240502 14:05:04.827189 30908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } }
I20240502 14:05:04.827879 30908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:04.828167 30908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293fb3dde84247f88a9c33df60fb0130, State: Initialized, Role: FOLLOWER
I20240502 14:05:04.828897 30908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } }
I20240502 14:05:04.829409 30908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:04.829672 30908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:04.829970 30908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:04.833755 30908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } }
I20240502 14:05:04.834434 30908 leader_election.cc:302] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293fb3dde84247f88a9c33df60fb0130; no voters:
I20240502 14:05:04.836115 30908 leader_election.cc:288] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:04.836517 30913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:04.838620 30913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [term 1 LEADER]: Becoming Leader. State: Replica: 293fb3dde84247f88a9c33df60fb0130, State: Running, Role: LEADER
I20240502 14:05:04.839447 30913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } }
I20240502 14:05:04.840304 30908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:04.850999 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "293fb3dde84247f88a9c33df60fb0130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } } }
I20240502 14:05:04.852324 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:04.853381 30915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 293fb3dde84247f88a9c33df60fb0130. Latest consensus state: current_term: 1 leader_uuid: "293fb3dde84247f88a9c33df60fb0130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293fb3dde84247f88a9c33df60fb0130" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34649 } } }
I20240502 14:05:04.854348 30915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:04.857000 30921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:04.869043 30921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:04.886868 30921 catalog_manager.cc:1347] Generated new cluster ID: b666538c11904d69b554aadbc0d26fc7
I20240502 14:05:04.887182 30921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:04.916299 30921 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:04.917697 30921 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:04.934252 30921 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 293fb3dde84247f88a9c33df60fb0130: Generated new TSK 0
I20240502 14:05:04.935148 30921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:05:04.996989 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:05.285318 30934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:05.285835 30934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:05.286520 30934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:05.318157 30934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:05.318540 30934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:05.319485 30934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:05:05.353510 30934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:05.354691 30934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:05.355686 30934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:05.372175 30940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:05.373569 30943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:05.373059 30941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:06.445583 30942 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:05:06.445811 30934 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:05:06.449752 30934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:06.452580 30934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:06.453998 30934 hybrid_clock.cc:648] HybridClock initialized: now 1714658706453929 us; error 75 us; skew 500 ppm
I20240502 14:05:06.455035 30934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:06 dist-test-slave-dlb8 krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658706, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:06.462165 30934 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:05:06.468225 30934 webserver.cc:460] Webserver started at http://127.27.174.65:33793/ using document root <none> and password file <none>
I20240502 14:05:06.469604 30934 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:06.469897 30934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:06.470563 30934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:06.478121 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "b9819ca5a6204ce39c71986181de07f2"
format_stamp: "Formatted at 2024-05-02 14:05:06 on dist-test-slave-dlb8"
server_key: "c80867fdbcb1be6c323ac205bb7f121b"
server_key_iv: "86915919bc73b972878b38f51dacb556"
server_key_version: "encryptionkey@0"
I20240502 14:05:06.479669 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "b9819ca5a6204ce39c71986181de07f2"
format_stamp: "Formatted at 2024-05-02 14:05:06 on dist-test-slave-dlb8"
server_key: "c80867fdbcb1be6c323ac205bb7f121b"
server_key_iv: "86915919bc73b972878b38f51dacb556"
server_key_version: "encryptionkey@0"
I20240502 14:05:06.488402 30934 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20240502 14:05:06.495453 30951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:06.496685 30934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:05:06.497045 30934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "b9819ca5a6204ce39c71986181de07f2"
format_stamp: "Formatted at 2024-05-02 14:05:06 on dist-test-slave-dlb8"
server_key: "c80867fdbcb1be6c323ac205bb7f121b"
server_key_iv: "86915919bc73b972878b38f51dacb556"
server_key_version: "encryptionkey@0"
I20240502 14:05:06.497488 30934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:06.549055 30934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:06.554147 30934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:06.554596 30934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:06.556942 30934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:06.560922 30934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:06.561111 30934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:06.561381 30934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:06.561540 30934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:06.691668 30934 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44223
I20240502 14:05:06.691778 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44223 every 8 connection(s)
I20240502 14:05:06.694408 30934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:05:06.704339 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 30934
I20240502 14:05:06.704754 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:05:06.705699 28345 external_mini_cluster.cc:1428] Setting key e2224dd7969b94461810e82f91553831
May 02 14:05:06 dist-test-slave-dlb8 krb5kdc[30826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658706, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20240502 14:05:06.728672 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:06.698561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35673 (local address 127.27.174.126:34649)
0502 14:05:06.699656 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:06.699674 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:06.703349 (+ 3675us) server_negotiation.cc:407] Connection header received
0502 14:05:06.708211 (+ 4862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:06.708230 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:06.708386 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:06.708521 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:06.711143 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:06.712547 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:06.713973 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:06.714615 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:06.718504 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:06.718577 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:06.718594 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:06.718689 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:06.722447 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:06.723584 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:06.723609 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:06.723624 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:06.723770 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:06.724376 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:06.724397 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:06.724410 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:06.724802 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:06.725022 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:06.725999 (+ 977us) server_negotiation.cc:299] Negotiation successful
0502 14:05:06.726493 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:05:06.729219 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:06.699073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34649 (local address 127.27.174.65:35673)
0502 14:05:06.703112 (+ 4039us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:06.703178 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:06.707857 (+ 4679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:06.708820 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:06.708862 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:06.709837 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:06.710863 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:06.710903 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:06.712805 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:06.712827 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:06.713722 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:06.713763 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:06.714087 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:06.714935 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:06.714998 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:06.718228 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:06.722685 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:06.722711 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:06.722734 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:06.723368 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:06.723937 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:06.723958 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:06.723972 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:06.724170 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:06.725000 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:06.725030 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:06.725433 (+ 403us) client_negotiation.cc:769] Sending connection context
0502 14:05:06.725761 (+ 328us) client_negotiation.cc:240] Negotiation successful
0502 14:05:06.726397 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3523,"thread_start_us":3060,"threads_started":1}
I20240502 14:05:06.731326 31064 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34649
I20240502 14:05:06.731726 31064 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:06.732707 31064 heartbeater.cc:511] Master 127.27.174.126:34649 requested a full tablet report, sending...
I20240502 14:05:06.735603 30873 ts_manager.cc:194] Registered new tserver with Master: b9819ca5a6204ce39c71986181de07f2 (127.27.174.65:44223)
I20240502 14:05:06.737505 30873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:35673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:05:06.756069 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:07.050956 31070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:07.051437 31070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:07.052035 31070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:07.081053 31070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:07.081435 31070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:07.082401 31070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:05:07.115849 31070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:07.117005 31070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:07.118000 31070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:07.133029 31077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:07.741468 31064 heartbeater.cc:503] Master 127.27.174.126:34649 was elected leader, sending a full tablet report...
W20240502 14:05:07.133837 31079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:07.135039 31076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:07.134981 31070 server_base.cc:1034] running on GCE node
I20240502 14:05:08.171972 31070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:08.174686 31070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:08.176082 31070 hybrid_clock.cc:648] HybridClock initialized: now 1714658708176055 us; error 47 us; skew 500 ppm
I20240502 14:05:08.176784 31070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:08 dist-test-slave-dlb8 krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:08.187788 31070 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:05:08.193169 31070 webserver.cc:460] Webserver started at http://127.27.174.66:36195/ using document root <none> and password file <none>
I20240502 14:05:08.193985 31070 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:08.194279 31070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:08.194689 31070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:08.199411 31070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "cc2763fb17d1453ea77feccaf20df997"
format_stamp: "Formatted at 2024-05-02 14:05:08 on dist-test-slave-dlb8"
server_key: "752980a91875062967dc134b15cdfea9"
server_key_iv: "b4e95faeea727be0beab9e586da18964"
server_key_version: "encryptionkey@0"
I20240502 14:05:08.200570 31070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "cc2763fb17d1453ea77feccaf20df997"
format_stamp: "Formatted at 2024-05-02 14:05:08 on dist-test-slave-dlb8"
server_key: "752980a91875062967dc134b15cdfea9"
server_key_iv: "b4e95faeea727be0beab9e586da18964"
server_key_version: "encryptionkey@0"
I20240502 14:05:08.207300 31070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:05:08.213121 31088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:08.214207 31070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:05:08.214517 31070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "cc2763fb17d1453ea77feccaf20df997"
format_stamp: "Formatted at 2024-05-02 14:05:08 on dist-test-slave-dlb8"
server_key: "752980a91875062967dc134b15cdfea9"
server_key_iv: "b4e95faeea727be0beab9e586da18964"
server_key_version: "encryptionkey@0"
I20240502 14:05:08.214953 31070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:08.279018 31070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:08.284410 31070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:08.284832 31070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:08.287159 31070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:08.291074 31070 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:08.291277 31070 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:08.291514 31070 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:08.291663 31070 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:08.421692 31070 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45725
I20240502 14:05:08.421802 31200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45725 every 8 connection(s)
I20240502 14:05:08.424252 31070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:05:08.427209 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31070
I20240502 14:05:08.428000 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:05:08.429041 28345 external_mini_cluster.cc:1428] Setting key 5f03aa83325f2c034df639613fe7d483
May 02 14:05:08 dist-test-slave-dlb8 krb5kdc[30826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658708, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:05:08.456086 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:08.428673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38005 (local address 127.27.174.126:34649)
0502 14:05:08.430344 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:08.430365 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:08.432077 (+ 1712us) server_negotiation.cc:407] Connection header received
0502 14:05:08.436789 (+ 4712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:08.436810 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:08.436976 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:08.437122 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:08.439501 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:08.440473 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:08.441987 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:08.442550 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:08.446815 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:08.446925 (+ 110us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:08.446953 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:08.447058 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:08.450740 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:08.451832 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:08.451856 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:08.451870 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:08.451999 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:08.452651 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:08.452688 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:08.452702 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:08.453077 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:08.453261 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:08.454496 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0502 14:05:08.454815 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":529,"threads_started":1}
I20240502 14:05:08.457911 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:08.428840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34649 (local address 127.27.174.66:38005)
0502 14:05:08.431871 (+ 3031us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:08.431933 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:08.436414 (+ 4481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:08.437421 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:08.437449 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:08.438288 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:08.439244 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:08.439284 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:08.440710 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:08.440736 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:08.441719 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:08.441780 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:08.442191 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:08.443187 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:08.443250 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:08.446510 (+ 3260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:08.450973 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:08.450994 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:08.451017 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:08.451650 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:08.452133 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:08.452157 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:08.452183 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:08.452415 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:08.453304 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:08.453340 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:08.453873 (+ 533us) client_negotiation.cc:769] Sending connection context
0502 14:05:08.454347 (+ 474us) client_negotiation.cc:240] Negotiation successful
0502 14:05:08.454960 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2382,"thread_start_us":1784,"threads_started":1}
I20240502 14:05:08.459899 31201 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34649
I20240502 14:05:08.460407 31201 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:08.461894 31201 heartbeater.cc:511] Master 127.27.174.126:34649 requested a full tablet report, sending...
I20240502 14:05:08.464283 30873 ts_manager.cc:194] Registered new tserver with Master: cc2763fb17d1453ea77feccaf20df997 (127.27.174.66:45725)
I20240502 14:05:08.465448 30873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:38005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:05:08.479635 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:08.780593 31207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:08.781114 31207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:08.781687 31207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:08.810060 31207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:08.810482 31207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:08.811393 31207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:05:08.844151 31207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:08.845319 31207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:08.846417 31207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:08.861505 31213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:09.468895 31201 heartbeater.cc:503] Master 127.27.174.126:34649 was elected leader, sending a full tablet report...
W20240502 14:05:08.862370 31216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:08.863247 31207 server_base.cc:1034] running on GCE node
W20240502 14:05:08.863329 31214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:09.884707 31207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:09.887392 31207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:09.888792 31207 hybrid_clock.cc:648] HybridClock initialized: now 1714658709888762 us; error 58 us; skew 500 ppm
I20240502 14:05:09.889521 31207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:09 dist-test-slave-dlb8 krb5kdc[30826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658709, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:09.895010 31207 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:05:09.900269 31207 webserver.cc:460] Webserver started at http://127.27.174.67:39079/ using document root <none> and password file <none>
I20240502 14:05:09.901166 31207 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:09.901383 31207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:09.901824 31207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:09.906682 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "0c7f4136572d42eea8982931bbd3ecca"
format_stamp: "Formatted at 2024-05-02 14:05:09 on dist-test-slave-dlb8"
server_key: "073d8966352b673fca9a4b9e5831bbad"
server_key_iv: "36bc0bb66419b23a746d6ae4954bffe0"
server_key_version: "encryptionkey@0"
I20240502 14:05:09.907733 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "0c7f4136572d42eea8982931bbd3ecca"
format_stamp: "Formatted at 2024-05-02 14:05:09 on dist-test-slave-dlb8"
server_key: "073d8966352b673fca9a4b9e5831bbad"
server_key_iv: "36bc0bb66419b23a746d6ae4954bffe0"
server_key_version: "encryptionkey@0"
I20240502 14:05:09.914338 31207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20240502 14:05:09.919325 31224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:09.920331 31207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:05:09.920629 31207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "0c7f4136572d42eea8982931bbd3ecca"
format_stamp: "Formatted at 2024-05-02 14:05:09 on dist-test-slave-dlb8"
server_key: "073d8966352b673fca9a4b9e5831bbad"
server_key_iv: "36bc0bb66419b23a746d6ae4954bffe0"
server_key_version: "encryptionkey@0"
I20240502 14:05:09.920967 31207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:09.977777 31207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:09.983047 31207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:09.983443 31207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:09.985641 31207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:09.989192 31207 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:09.989375 31207 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:09.989606 31207 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:09.989737 31207 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:10.112771 31207 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46711
I20240502 14:05:10.112857 31336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46711 every 8 connection(s)
I20240502 14:05:10.115105 31207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:05:10.118726 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31207
I20240502 14:05:10.119189 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:05:10.120374 28345 external_mini_cluster.cc:1428] Setting key 2d17a34c1f014d15e0b061b4721b9187
May 02 14:05:10 dist-test-slave-dlb8 krb5kdc[30826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658709, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:10.146440 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:10.121502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:43975 (local address 127.27.174.126:34649)
0502 14:05:10.124054 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:10.124076 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:10.124131 (+ 55us) server_negotiation.cc:407] Connection header received
0502 14:05:10.127966 (+ 3835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:10.127989 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:10.128151 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:10.128291 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:10.130572 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:10.131546 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:10.132816 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:10.133314 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:10.137529 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:10.137597 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:10.137613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:10.137702 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:10.141718 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:10.142780 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:10.142804 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:10.142817 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:10.142948 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:10.143428 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:10.143444 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:10.143456 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:10.143807 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:10.143971 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:10.144757 (+ 786us) server_negotiation.cc:299] Negotiation successful
0502 14:05:10.145110 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1676,"threads_started":1}
I20240502 14:05:10.147030 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:10.120320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34649 (local address 127.27.174.67:43975)
0502 14:05:10.123201 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:10.123315 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:10.127638 (+ 4323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:10.128565 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:10.128590 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:10.129339 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:10.130307 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:10.130352 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:10.131776 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:10.131798 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:10.132606 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:10.132639 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:10.132981 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:10.133976 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:10.134072 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:10.137287 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:10.141924 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:10.141945 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:10.141970 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:10.142614 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:10.143089 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:10.143108 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:10.143124 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:10.143298 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:10.143942 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:10.143961 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:10.144302 (+ 341us) client_negotiation.cc:769] Sending connection context
0502 14:05:10.144578 (+ 276us) client_negotiation.cc:240] Negotiation successful
0502 14:05:10.144981 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2180,"thread_start_us":1661,"threads_started":1}
I20240502 14:05:10.148581 31337 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34649
I20240502 14:05:10.148952 31337 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:10.150092 31337 heartbeater.cc:511] Master 127.27.174.126:34649 requested a full tablet report, sending...
I20240502 14:05:10.152302 30873 ts_manager.cc:194] Registered new tserver with Master: 0c7f4136572d42eea8982931bbd3ecca (127.27.174.67:46711)
I20240502 14:05:10.153636 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:05:10.153831 30873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:43975
W20240502 14:05:10.199910 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:05:10.194660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.27.174.126:34649)
0502 14:05:10.195175 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:10.195194 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:10.196384 (+ 1190us) server_negotiation.cc:407] Connection header received
0502 14:05:10.196792 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:10.196811 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:10.196995 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:10.197140 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:10.198926 (+ 1786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48910: BlockingRecv error: recv got EOF from 127.0.0.1:48910 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20240502 14:05:10.206696 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30934
I20240502 14:05:10.228225 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31070
I20240502 14:05:10.249456 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31207
I20240502 14:05:10.272079 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 30839
2024-05-02T14:05:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (8285 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:10 dist-test-slave-dlb8 krb5kdc[31350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:10 dist-test-slave-dlb8 krb5kdc[31350](info): set up 2 sockets
May 02 14:05:10 dist-test-slave-dlb8 krb5kdc[31350](info): commencing operation
krb5kdc: starting...
W20240502 14:05:11.251571 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:11 dist-test-slave-dlb8 krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:11.456079 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40189
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:11.742552 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:11.743096 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:11.743624 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:11.771822 31363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:11.772110 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:11.772326 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:11.772505 31363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:11.772677 31363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:11.805537 31363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:11.806649 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:11.807587 31363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:11.821610 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:11.822342 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:11.824654 31363 server_base.cc:1034] running on GCE node
W20240502 14:05:11.823274 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:12.873847 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:12.876358 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:12.877688 31363 hybrid_clock.cc:648] HybridClock initialized: now 1714658712877652 us; error 54 us; skew 500 ppm
I20240502 14:05:12.878461 31363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:12 dist-test-slave-dlb8 krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658712, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:12.884122 31363 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:05:12.888873 31363 webserver.cc:460] Webserver started at http://127.27.174.126:37777/ using document root <none> and password file <none>
I20240502 14:05:12.889695 31363 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:12.889891 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:12.890429 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:12.895148 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "0a947ad21ff24537b006aedc6e807927"
format_stamp: "Formatted at 2024-05-02 14:05:12 on dist-test-slave-dlb8"
server_key: "21d9885612f0102ce43d287bfaf52fa1"
server_key_iv: "0dc477fe7ac243c3900a0dad89bbb113"
server_key_version: "encryptionkey@0"
I20240502 14:05:12.896188 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "0a947ad21ff24537b006aedc6e807927"
format_stamp: "Formatted at 2024-05-02 14:05:12 on dist-test-slave-dlb8"
server_key: "21d9885612f0102ce43d287bfaf52fa1"
server_key_iv: "0dc477fe7ac243c3900a0dad89bbb113"
server_key_version: "encryptionkey@0"
I20240502 14:05:12.902940 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:05:12.907858 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:12.908841 31363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:05:12.909137 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "0a947ad21ff24537b006aedc6e807927"
format_stamp: "Formatted at 2024-05-02 14:05:12 on dist-test-slave-dlb8"
server_key: "21d9885612f0102ce43d287bfaf52fa1"
server_key_iv: "0dc477fe7ac243c3900a0dad89bbb113"
server_key_version: "encryptionkey@0"
I20240502 14:05:12.909480 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:12.961416 31363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:12.966198 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:12.966588 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:13.030076 31363 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:40189
I20240502 14:05:13.030200 31431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:40189 every 8 connection(s)
I20240502 14:05:13.032697 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:13.033593 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31363
I20240502 14:05:13.034035 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:13.035009 28345 external_mini_cluster.cc:1428] Setting key 0bf3a27c38da3a06ce170251d0df058b
I20240502 14:05:13.039196 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:05:13 dist-test-slave-dlb8 krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:13.060570 31432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927: Bootstrap starting.
I20240502 14:05:13.066329 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:13.037888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.27.174.126:40189)
0502 14:05:13.040319 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:13.040346 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:13.040403 (+ 57us) server_negotiation.cc:407] Connection header received
0502 14:05:13.044736 (+ 4333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:13.044775 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:13.045111 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:13.045511 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:13.048075 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:13.049326 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:13.050693 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:13.051291 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:13.055970 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:13.056061 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:13.056089 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:13.056198 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:13.060180 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:13.061554 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:13.061575 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:13.061604 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:13.061798 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:13.062422 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:13.062442 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:13.062455 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:13.063008 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:13.063336 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:13.064010 (+ 674us) server_negotiation.cc:299] Negotiation successful
0502 14:05:13.064428 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1830,"thread_start_us":1442,"threads_started":1}
I20240502 14:05:13.066416 31432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:13.068184 31432 log.cc:824] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:13.070792 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:05:13.073802 31432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927: No bootstrap required, opened a new log
I20240502 14:05:13.098910 31432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } }
I20240502 14:05:13.099551 31432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:13.099785 31432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a947ad21ff24537b006aedc6e807927, State: Initialized, Role: FOLLOWER
I20240502 14:05:13.100555 31432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } }
I20240502 14:05:13.101034 31432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:13.101316 31432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:13.101621 31432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:13.105525 31432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } }
I20240502 14:05:13.106242 31432 leader_election.cc:302] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a947ad21ff24537b006aedc6e807927; no voters:
I20240502 14:05:13.108330 31432 leader_election.cc:288] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:13.108639 31439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:13.112506 31439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [term 1 LEADER]: Becoming Leader. State: Replica: 0a947ad21ff24537b006aedc6e807927, State: Running, Role: LEADER
I20240502 14:05:13.113421 31439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } }
I20240502 14:05:13.113643 31432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:13.122923 31441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a947ad21ff24537b006aedc6e807927. Latest consensus state: current_term: 1 leader_uuid: "0a947ad21ff24537b006aedc6e807927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } } }
I20240502 14:05:13.123729 31441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:13.125247 31440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a947ad21ff24537b006aedc6e807927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a947ad21ff24537b006aedc6e807927" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40189 } } }
I20240502 14:05:13.126107 31440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:13.130847 31451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:13.143106 31451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:13.144269 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:13.117387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.27.174.126:40189)
0502 14:05:13.118262 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:13.118294 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:13.129737 (+ 11443us) server_negotiation.cc:407] Connection header received
0502 14:05:13.130342 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:13.130425 (+ 83us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:13.130671 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:13.130841 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:13.133808 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:13.135063 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:13.136990 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:13.137539 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:13.139327 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:13.139420 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:13.139453 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:13.140469 (+ 1016us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:13.140614 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:13.140696 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:13.140856 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:13.142083 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0502 14:05:13.142276 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20240502 14:05:13.161196 31451 catalog_manager.cc:1347] Generated new cluster ID: 7293e38fff734eddb0b6b63394b82938
I20240502 14:05:13.161525 31451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:13.183745 31451 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:13.185544 31451 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:13.199244 31451 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0a947ad21ff24537b006aedc6e807927: Generated new TSK 0
I20240502 14:05:13.200035 31451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:13.229641 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31363
2024-05-02T14:05:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2945 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:13 dist-test-slave-dlb8 krb5kdc[31465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:13 dist-test-slave-dlb8 krb5kdc[31465](info): set up 2 sockets
May 02 14:05:13 dist-test-slave-dlb8 krb5kdc[31465](info): commencing operation
krb5kdc: starting...
W20240502 14:05:14.179500 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.906s user 0.009s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:14 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:14.379204 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34283
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:14.664335 31478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:14.664938 31478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:14.665520 31478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:14.698438 31478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:14.698774 31478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:14.699028 31478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:14.699268 31478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:14.699479 31478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:14.733805 31478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:14.735002 31478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:14.736011 31478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:14.753088 31487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:14.753108 31484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:14.753993 31485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:14.755016 31478 server_base.cc:1034] running on GCE node
I20240502 14:05:15.796360 31478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:15.799521 31478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:15.800959 31478 hybrid_clock.cc:648] HybridClock initialized: now 1714658715800921 us; error 58 us; skew 500 ppm
I20240502 14:05:15.801752 31478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:15 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658715, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:15.807376 31478 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:05:15.812786 31478 webserver.cc:460] Webserver started at http://127.27.174.126:37865/ using document root <none> and password file <none>
I20240502 14:05:15.813663 31478 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:15.813868 31478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:15.814333 31478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:15.819202 31478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "3163a3755e7a43b1941c3ed25b3a7206"
format_stamp: "Formatted at 2024-05-02 14:05:15 on dist-test-slave-dlb8"
server_key: "dd5f4d537a4a6234696da547f2094add"
server_key_iv: "8c571275f0f80aa9a861a5c59010ea6d"
server_key_version: "encryptionkey@0"
I20240502 14:05:15.820255 31478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "3163a3755e7a43b1941c3ed25b3a7206"
format_stamp: "Formatted at 2024-05-02 14:05:15 on dist-test-slave-dlb8"
server_key: "dd5f4d537a4a6234696da547f2094add"
server_key_iv: "8c571275f0f80aa9a861a5c59010ea6d"
server_key_version: "encryptionkey@0"
I20240502 14:05:15.827230 31478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 14:05:15.832821 31495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:15.833896 31478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240502 14:05:15.834276 31478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "3163a3755e7a43b1941c3ed25b3a7206"
format_stamp: "Formatted at 2024-05-02 14:05:15 on dist-test-slave-dlb8"
server_key: "dd5f4d537a4a6234696da547f2094add"
server_key_iv: "8c571275f0f80aa9a861a5c59010ea6d"
server_key_version: "encryptionkey@0"
I20240502 14:05:15.834636 31478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:15.886408 31478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:15.891899 31478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:15.892400 31478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:15.957793 31478 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34283
I20240502 14:05:15.957859 31546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34283 every 8 connection(s)
I20240502 14:05:15.960259 31478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:15.965155 31547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:15.968338 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31478
I20240502 14:05:15.968768 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:15.969792 28345 external_mini_cluster.cc:1428] Setting key f77567795060481e43478f6dd82360f7
I20240502 14:05:15.986456 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206: Bootstrap starting.
May 02 14:05:15 dist-test-slave-dlb8 krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:15.991627 31547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:15.993854 31547 log.cc:824] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:15.999605 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206: No bootstrap required, opened a new log
I20240502 14:05:16.003857 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:15.972428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.27.174.126:34283)
0502 14:05:15.975121 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:15.975148 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:15.975198 (+ 50us) server_negotiation.cc:407] Connection header received
0502 14:05:15.978861 (+ 3663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:15.978897 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:15.979223 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:15.979549 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:15.982939 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:15.984324 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:15.985968 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:15.986391 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:15.990895 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:15.991021 (+ 126us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:15.991069 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:15.991211 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:15.996660 (+ 5449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:15.997923 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:15.997958 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:15.997999 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:15.998233 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:15.998723 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:15.998748 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:15.998768 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:15.999495 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:15.999949 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:16.000498 (+ 549us) server_negotiation.cc:299] Negotiation successful
0502 14:05:16.001073 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2191,"thread_start_us":1772,"threads_started":1}
I20240502 14:05:16.019532 31547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } }
I20240502 14:05:16.020206 31547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:16.020398 31547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3163a3755e7a43b1941c3ed25b3a7206, State: Initialized, Role: FOLLOWER
I20240502 14:05:16.021081 31547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } }
I20240502 14:05:16.021564 31547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:16.021816 31547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:16.022081 31547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:16.025910 31547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } }
I20240502 14:05:16.026594 31547 leader_election.cc:302] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3163a3755e7a43b1941c3ed25b3a7206; no voters:
I20240502 14:05:16.028085 31547 leader_election.cc:288] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:16.028563 31552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:16.030812 31552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [term 1 LEADER]: Becoming Leader. State: Replica: 3163a3755e7a43b1941c3ed25b3a7206, State: Running, Role: LEADER
I20240502 14:05:16.031710 31552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } }
I20240502 14:05:16.032932 31547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:16.042766 31554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3163a3755e7a43b1941c3ed25b3a7206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } } }
I20240502 14:05:16.042996 31553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3163a3755e7a43b1941c3ed25b3a7206. Latest consensus state: current_term: 1 leader_uuid: "3163a3755e7a43b1941c3ed25b3a7206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3163a3755e7a43b1941c3ed25b3a7206" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34283 } } }
I20240502 14:05:16.043505 31554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:16.043689 31553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:16.047240 31562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:16.057600 31562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:16.072726 31562 catalog_manager.cc:1347] Generated new cluster ID: 02e93856e6254c538bfe6a94cd92bdc0
I20240502 14:05:16.073045 31562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:16.106102 31562 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:16.107522 31562 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:16.119659 31562 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3163a3755e7a43b1941c3ed25b3a7206: Generated new TSK 0
I20240502 14:05:16.120704 31562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:05:16.192071 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:16.492305 31573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:16.492807 31573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:16.493395 31573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:16.524214 31573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:16.524574 31573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:16.525461 31573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:05:16.559921 31573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:16.561043 31573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:16.562119 31573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:16.582473 31580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:16.583943 31579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:17.905615 31581 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1327 milliseconds
W20240502 14:05:17.907765 31582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:17.910535 31573 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.336s user 0.398s sys 0.810s
W20240502 14:05:17.910809 31573 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.336s user 0.398s sys 0.810s
I20240502 14:05:17.911058 31573 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:05:17.912971 31573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:17.918660 31573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:17.920096 31573 hybrid_clock.cc:648] HybridClock initialized: now 1714658717920043 us; error 74 us; skew 500 ppm
I20240502 14:05:17.920888 31573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:17 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:17.927388 31573 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:05:17.935176 31573 webserver.cc:460] Webserver started at http://127.27.174.65:38059/ using document root <none> and password file <none>
I20240502 14:05:17.936084 31573 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:17.936300 31573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:17.936753 31573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:17.941782 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "316e57b695724985b4ef223828f4070c"
format_stamp: "Formatted at 2024-05-02 14:05:17 on dist-test-slave-dlb8"
server_key: "7dfcd9e309c45467f769f29e7e56e2d3"
server_key_iv: "1f6937e6a79f9e879dfe27100d455727"
server_key_version: "encryptionkey@0"
I20240502 14:05:17.942963 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "316e57b695724985b4ef223828f4070c"
format_stamp: "Formatted at 2024-05-02 14:05:17 on dist-test-slave-dlb8"
server_key: "7dfcd9e309c45467f769f29e7e56e2d3"
server_key_iv: "1f6937e6a79f9e879dfe27100d455727"
server_key_version: "encryptionkey@0"
I20240502 14:05:17.951911 31573 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20240502 14:05:17.973062 31591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:17.974288 31573 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.015s sys 0.001s
I20240502 14:05:17.974594 31573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "316e57b695724985b4ef223828f4070c"
format_stamp: "Formatted at 2024-05-02 14:05:17 on dist-test-slave-dlb8"
server_key: "7dfcd9e309c45467f769f29e7e56e2d3"
server_key_iv: "1f6937e6a79f9e879dfe27100d455727"
server_key_version: "encryptionkey@0"
I20240502 14:05:17.974973 31573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:18.045758 31573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:18.051474 31573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:18.051929 31573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:18.054458 31573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:18.058966 31573 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:18.059180 31573 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:18.059414 31573 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:18.059568 31573 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:18.218351 31573 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:45815
I20240502 14:05:18.218565 31703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:45815 every 8 connection(s)
I20240502 14:05:18.220907 31573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:05:18.229478 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31573
I20240502 14:05:18.230243 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:05:18.231812 28345 external_mini_cluster.cc:1428] Setting key 57d6f3c923ee7e4ddd43d8b4547cc8f9
May 02 14:05:18 dist-test-slave-dlb8 krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:05:18.262712 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:18.225277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:52603 (local address 127.27.174.126:34283)
0502 14:05:18.226558 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:18.226575 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:18.227978 (+ 1403us) server_negotiation.cc:407] Connection header received
0502 14:05:18.232586 (+ 4608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:18.232612 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:18.232763 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:18.232904 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:18.235875 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:18.237054 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:18.238448 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:18.238944 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:18.249951 (+ 11007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:18.250086 (+ 135us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:18.250121 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:18.250281 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:18.255717 (+ 5436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:18.257242 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:18.257261 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:18.257274 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:18.257418 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:18.258191 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:18.258212 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:18.258224 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:18.258619 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:18.258896 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:18.259992 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0502 14:05:18.260434 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":538,"threads_started":1}
I20240502 14:05:18.263489 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:18.225547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34283 (local address 127.27.174.65:52603)
0502 14:05:18.227741 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:18.227801 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:18.232237 (+ 4436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:18.233260 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:18.233287 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:18.234337 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:18.235555 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:18.235612 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:18.237413 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:18.237432 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:18.238243 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:18.238276 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:18.238620 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:18.245269 (+ 6649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:18.245374 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:18.249582 (+ 4208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:18.255991 (+ 6409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:18.256023 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:18.256058 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:18.256979 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:18.257574 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:18.257605 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:18.257627 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:18.257903 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:18.258882 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:18.258912 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:18.259324 (+ 412us) client_negotiation.cc:769] Sending connection context
0502 14:05:18.259764 (+ 440us) client_negotiation.cc:240] Negotiation successful
0502 14:05:18.260366 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"thread_start_us":1243,"threads_started":1}
I20240502 14:05:18.265659 31704 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34283
I20240502 14:05:18.266106 31704 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:18.267302 31704 heartbeater.cc:511] Master 127.27.174.126:34283 requested a full tablet report, sending...
I20240502 14:05:18.270727 31512 ts_manager.cc:194] Registered new tserver with Master: 316e57b695724985b4ef223828f4070c (127.27.174.65:45815)
I20240502 14:05:18.273257 31512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:52603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:05:18.285961 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:18.574995 31710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:18.575480 31710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:18.576048 31710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:18.604452 31710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:18.604779 31710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:18.605623 31710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:05:18.639776 31710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:18.640844 31710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:18.641844 31710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:18.657109 31717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:19.277374 31704 heartbeater.cc:503] Master 127.27.174.126:34283 was elected leader, sending a full tablet report...
W20240502 14:05:18.658450 31716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:18.659884 31710 server_base.cc:1034] running on GCE node
W20240502 14:05:18.658680 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:19.694823 31710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:19.696959 31710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:19.698290 31710 hybrid_clock.cc:648] HybridClock initialized: now 1714658719698253 us; error 55 us; skew 500 ppm
I20240502 14:05:19.699003 31710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:19 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658719, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:19.704572 31710 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:05:19.709297 31710 webserver.cc:460] Webserver started at http://127.27.174.66:37207/ using document root <none> and password file <none>
I20240502 14:05:19.710135 31710 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:19.710338 31710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:19.710737 31710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:19.715615 31710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c12561e00a0a469795107be528334fcc"
format_stamp: "Formatted at 2024-05-02 14:05:19 on dist-test-slave-dlb8"
server_key: "606fc595c7aee1ca22649c2e67a74618"
server_key_iv: "c5e3351e00fccc6698d0507127fc94d7"
server_key_version: "encryptionkey@0"
I20240502 14:05:19.716733 31710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c12561e00a0a469795107be528334fcc"
format_stamp: "Formatted at 2024-05-02 14:05:19 on dist-test-slave-dlb8"
server_key: "606fc595c7aee1ca22649c2e67a74618"
server_key_iv: "c5e3351e00fccc6698d0507127fc94d7"
server_key_version: "encryptionkey@0"
I20240502 14:05:19.723727 31710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20240502 14:05:19.728958 31727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:19.729951 31710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:05:19.730320 31710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "c12561e00a0a469795107be528334fcc"
format_stamp: "Formatted at 2024-05-02 14:05:19 on dist-test-slave-dlb8"
server_key: "606fc595c7aee1ca22649c2e67a74618"
server_key_iv: "c5e3351e00fccc6698d0507127fc94d7"
server_key_version: "encryptionkey@0"
I20240502 14:05:19.730696 31710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:19.782418 31710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:19.787039 31710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:19.787465 31710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:19.789770 31710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:19.793561 31710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:19.793746 31710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:19.793972 31710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:19.794131 31710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 14:05:19.916631 31710 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:43975
I20240502 14:05:19.916725 31839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:43975 every 8 connection(s)
I20240502 14:05:19.918967 31710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:05:19.920152 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31710
I20240502 14:05:19.920639 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:05:19.921818 28345 external_mini_cluster.cc:1428] Setting key 4a45efbfed84cbe0084eb6044d8d6c32
May 02 14:05:19 dist-test-slave-dlb8 krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658719, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:05:19.953034 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:19.924218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34283 (local address 127.27.174.66:41983)
0502 14:05:19.927899 (+ 3681us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:19.927961 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:19.932244 (+ 4283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:19.933186 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:19.933227 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:19.934442 (+ 1215us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:19.935747 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:19.935809 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:19.937123 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:19.937160 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:19.938464 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:19.938521 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:19.938922 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:19.939919 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:19.940018 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:19.943719 (+ 3701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:19.947602 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:19.947642 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:19.947685 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:19.948279 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:19.948910 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:19.948929 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:19.948942 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:19.949111 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:19.950181 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:19.950203 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:19.950552 (+ 349us) client_negotiation.cc:769] Sending connection context
0502 14:05:19.950854 (+ 302us) client_negotiation.cc:240] Negotiation successful
0502 14:05:19.951275 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3055,"thread_start_us":2555,"threads_started":1}
I20240502 14:05:19.953879 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:19.923905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:41983 (local address 127.27.174.126:34283)
0502 14:05:19.926810 (+ 2905us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:19.926836 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:19.928111 (+ 1275us) server_negotiation.cc:407] Connection header received
0502 14:05:19.932576 (+ 4465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:19.932595 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:19.932752 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:19.932890 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:19.936031 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:19.936880 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:19.938699 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:19.939022 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:19.943986 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:19.944059 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:19.944075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:19.944165 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:19.947378 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:19.948505 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:19.948542 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:19.948569 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:19.948740 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:19.949299 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:19.949337 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:19.949365 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:19.949948 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:19.950243 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:19.951140 (+ 897us) server_negotiation.cc:299] Negotiation successful
0502 14:05:19.951638 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2375,"thread_start_us":2140,"threads_started":1}
I20240502 14:05:19.955507 31840 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34283
I20240502 14:05:19.955943 31840 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:19.957080 31840 heartbeater.cc:511] Master 127.27.174.126:34283 requested a full tablet report, sending...
I20240502 14:05:19.959542 31512 ts_manager.cc:194] Registered new tserver with Master: c12561e00a0a469795107be528334fcc (127.27.174.66:43975)
I20240502 14:05:19.961360 31512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:41983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:05:19.976990 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:20.254294 31846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:20.254823 31846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:20.255401 31846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:20.283486 31846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:20.283878 31846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:20.284739 31846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:05:20.317176 31846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:20.318307 31846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:20.319293 31846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:20.335912 31852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:20.965939 31840 heartbeater.cc:503] Master 127.27.174.126:34283 was elected leader, sending a full tablet report...
W20240502 14:05:20.336699 31855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:20.337563 31853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:20.337700 31846 server_base.cc:1034] running on GCE node
I20240502 14:05:21.356035 31846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:21.358168 31846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:21.359489 31846 hybrid_clock.cc:648] HybridClock initialized: now 1714658721359475 us; error 46 us; skew 500 ppm
I20240502 14:05:21.360248 31846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:21.365875 31846 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:05:21.370970 31846 webserver.cc:460] Webserver started at http://127.27.174.67:32901/ using document root <none> and password file <none>
I20240502 14:05:21.371858 31846 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:21.372064 31846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:21.372520 31846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:21.377305 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "47c87abf25394da5a2119af8cbad0158"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-dlb8"
server_key: "41a3ef494979abcb3f9c7cc1732831cf"
server_key_iv: "6e4260648273f86d92f73fd1198c06e6"
server_key_version: "encryptionkey@0"
I20240502 14:05:21.378382 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "47c87abf25394da5a2119af8cbad0158"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-dlb8"
server_key: "41a3ef494979abcb3f9c7cc1732831cf"
server_key_iv: "6e4260648273f86d92f73fd1198c06e6"
server_key_version: "encryptionkey@0"
I20240502 14:05:21.385154 31846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 14:05:21.390311 31863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.391328 31846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:05:21.391647 31846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "47c87abf25394da5a2119af8cbad0158"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-dlb8"
server_key: "41a3ef494979abcb3f9c7cc1732831cf"
server_key_iv: "6e4260648273f86d92f73fd1198c06e6"
server_key_version: "encryptionkey@0"
I20240502 14:05:21.392030 31846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:21.457399 31846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:21.462688 31846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:21.463126 31846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:21.465432 31846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:21.469377 31846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:21.469583 31846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.469836 31846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:21.469995 31846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.592695 31846 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:40393
I20240502 14:05:21.592794 31975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:40393 every 8 connection(s)
I20240502 14:05:21.595219 31846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:05:21.600492 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 31846
I20240502 14:05:21.600960 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:05:21.601982 28345 external_mini_cluster.cc:1428] Setting key 6b89c563635381e115b656eb59021be5
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658721, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:21.626868 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:21.599101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56113 (local address 127.27.174.126:34283)
0502 14:05:21.602662 (+ 3561us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:21.602683 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:21.602733 (+ 50us) server_negotiation.cc:407] Connection header received
0502 14:05:21.607445 (+ 4712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:21.607480 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:21.607643 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:21.607800 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:21.610107 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.611046 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:21.612393 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.612930 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:21.616748 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:21.616821 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:21.616837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:21.616927 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:21.620632 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:21.621803 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:21.621836 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:21.621849 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:21.621988 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:21.622651 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:21.622670 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:21.622683 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:21.623077 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:21.623288 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:21.624279 (+ 991us) server_negotiation.cc:299] Negotiation successful
0502 14:05:21.624726 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2921,"thread_start_us":562,"threads_started":1}
I20240502 14:05:21.627480 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:21.599343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34283 (local address 127.27.174.67:56113)
0502 14:05:21.602103 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:21.602205 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:21.607108 (+ 4903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:21.608099 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:21.608128 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:21.608892 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:21.609826 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:21.609864 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.611292 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:21.611315 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:21.612145 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:21.612181 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.612562 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:21.613561 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:21.613632 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:21.616480 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:21.620860 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:21.620883 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:21.620906 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:21.621604 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:21.622218 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:21.622236 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:21.622254 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:21.622458 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:21.623288 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:21.623321 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:21.623733 (+ 412us) client_negotiation.cc:769] Sending connection context
0502 14:05:21.624068 (+ 335us) client_negotiation.cc:240] Negotiation successful
0502 14:05:21.624677 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"thread_start_us":1481,"threads_started":1}
I20240502 14:05:21.629143 31976 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34283
I20240502 14:05:21.629472 31976 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:21.630420 31976 heartbeater.cc:511] Master 127.27.174.126:34283 requested a full tablet report, sending...
I20240502 14:05:21.632555 31511 ts_manager.cc:194] Registered new tserver with Master: 47c87abf25394da5a2119af8cbad0158 (127.27.174.67:40393)
I20240502 14:05:21.634442 31511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:56113
I20240502 14:05:21.634963 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:21.699980 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:21.681236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.27.174.126:34283)
0502 14:05:21.681765 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:21.681793 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:21.682904 (+ 1111us) server_negotiation.cc:407] Connection header received
0502 14:05:21.683347 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:21.683366 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:21.683546 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:21.683675 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:21.685984 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.686803 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:21.688099 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:21.688431 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:21.692773 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:21.692843 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:21.692864 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:21.692955 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:21.696019 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:21.696922 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:21.696940 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:21.696952 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:21.697077 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:21.697481 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:21.697501 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:21.697513 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:21.697867 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:21.698176 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:21.698565 (+ 389us) server_negotiation.cc:299] Negotiation successful
0502 14:05:21.698849 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20240502 14:05:21.700902 31511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34488
I20240502 14:05:21.703517 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31573
I20240502 14:05:21.726675 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31710
I20240502 14:05:21.749786 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31846
I20240502 14:05:21.772480 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 31478
2024-05-02T14:05:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (8554 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31989](info): set up 2 sockets
May 02 14:05:21 dist-test-slave-dlb8 krb5kdc[31989](info): commencing operation
krb5kdc: starting...
W20240502 14:05:22.747848 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:22 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:22.945665 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46801
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:23.220762 32002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:23.221362 32002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:23.221980 32002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:23.250136 32002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:23.250449 32002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:23.250758 32002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:23.250989 32002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:23.251241 32002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:23.283987 32002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:23.285102 32002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:23.286134 32002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:23.300369 32008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:23.301976 32009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:23.303426 32002 server_base.cc:1034] running on GCE node
W20240502 14:05:23.302229 32011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:24.344970 32002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:24.347585 32002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:24.348922 32002 hybrid_clock.cc:648] HybridClock initialized: now 1714658724348868 us; error 68 us; skew 500 ppm
I20240502 14:05:24.349711 32002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:24 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658724, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:24.360917 32002 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:05:24.365993 32002 webserver.cc:460] Webserver started at http://127.27.174.126:37053/ using document root <none> and password file <none>
I20240502 14:05:24.366835 32002 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:24.367040 32002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:24.367492 32002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:24.372355 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "fba637aa4f8140b4a39663667b1a2fcb"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-dlb8"
server_key: "ca2cb1a897363aa34c333ee32f1f4c36"
server_key_iv: "e432c8735e01d67d16d371dbbb765d30"
server_key_version: "encryptionkey@0"
I20240502 14:05:24.373564 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "fba637aa4f8140b4a39663667b1a2fcb"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-dlb8"
server_key: "ca2cb1a897363aa34c333ee32f1f4c36"
server_key_iv: "e432c8735e01d67d16d371dbbb765d30"
server_key_version: "encryptionkey@0"
I20240502 14:05:24.380038 32002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:05:24.385380 32019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:24.386397 32002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:05:24.386706 32002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "fba637aa4f8140b4a39663667b1a2fcb"
format_stamp: "Formatted at 2024-05-02 14:05:24 on dist-test-slave-dlb8"
server_key: "ca2cb1a897363aa34c333ee32f1f4c36"
server_key_iv: "e432c8735e01d67d16d371dbbb765d30"
server_key_version: "encryptionkey@0"
I20240502 14:05:24.387039 32002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:24.435577 32002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:24.440745 32002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:24.441188 32002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:24.506027 32002 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46801
I20240502 14:05:24.506136 32070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46801 every 8 connection(s)
I20240502 14:05:24.508466 32002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:24.513172 32071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:24.516047 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32002
I20240502 14:05:24.516558 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:24.517939 28345 external_mini_cluster.cc:1428] Setting key e0069b82bd1c1089661914c90535661c
May 02 14:05:24 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:24.536592 32071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb: Bootstrap starting.
I20240502 14:05:24.543059 32071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:24.545053 32071 log.cc:824] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:24.550403 32071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb: No bootstrap required, opened a new log
I20240502 14:05:24.550972 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:24.520602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.27.174.126:46801)
0502 14:05:24.523161 (+ 2559us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:24.523187 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:24.523240 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:05:24.526812 (+ 3572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:24.526854 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:24.527184 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:24.527542 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:24.529943 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:24.531357 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:24.532720 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:24.533308 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:24.537658 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:24.537791 (+ 133us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:24.537828 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:24.537957 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:24.544438 (+ 6481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:24.545438 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:24.545466 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:24.545498 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:24.545650 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:24.546215 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:24.546248 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:24.546275 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:24.547093 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:24.547405 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:24.548117 (+ 712us) server_negotiation.cc:299] Negotiation successful
0502 14:05:24.548630 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2041,"thread_start_us":1606,"threads_started":1}
I20240502 14:05:24.568122 32071 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } }
I20240502 14:05:24.568751 32071 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:24.568938 32071 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba637aa4f8140b4a39663667b1a2fcb, State: Initialized, Role: FOLLOWER
I20240502 14:05:24.569473 32071 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } }
I20240502 14:05:24.569902 32071 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:24.570120 32071 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:24.570361 32071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:24.574067 32071 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } }
I20240502 14:05:24.574756 32071 leader_election.cc:302] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba637aa4f8140b4a39663667b1a2fcb; no voters:
I20240502 14:05:24.576264 32071 leader_election.cc:288] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:24.576612 32076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:24.579126 32076 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [term 1 LEADER]: Becoming Leader. State: Replica: fba637aa4f8140b4a39663667b1a2fcb, State: Running, Role: LEADER
I20240502 14:05:24.579883 32076 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } }
I20240502 14:05:24.580664 32071 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:24.589246 32078 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba637aa4f8140b4a39663667b1a2fcb. Latest consensus state: current_term: 1 leader_uuid: "fba637aa4f8140b4a39663667b1a2fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } } }
I20240502 14:05:24.590229 32078 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:24.591925 32077 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba637aa4f8140b4a39663667b1a2fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba637aa4f8140b4a39663667b1a2fcb" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46801 } } }
I20240502 14:05:24.592722 32077 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:24.595863 32083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:24.607062 32083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:24.621578 32083 catalog_manager.cc:1347] Generated new cluster ID: 2830c7027e7848c1a5225127a148f0d6
I20240502 14:05:24.621913 32083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:24.650717 32083 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:24.652864 32083 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:24.673318 32083 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fba637aa4f8140b4a39663667b1a2fcb: Generated new TSK 0
I20240502 14:05:24.674290 32083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:05:24.738999 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:25.027442 32097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:25.027935 32097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:25.028560 32097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:25.057929 32097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:25.058316 32097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:25.059126 32097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:05:25.091516 32097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:25.092661 32097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:25.093693 32097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:25.111030 32104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:25.111784 32106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:25.114135 32097 server_base.cc:1034] running on GCE node
W20240502 14:05:25.113096 32103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:26.162796 32097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:26.165491 32097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:26.166904 32097 hybrid_clock.cc:648] HybridClock initialized: now 1714658726166877 us; error 47 us; skew 500 ppm
I20240502 14:05:26.167656 32097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:26 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658726, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:26.173321 32097 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:05:26.178733 32097 webserver.cc:460] Webserver started at http://127.27.174.65:36075/ using document root <none> and password file <none>
I20240502 14:05:26.179574 32097 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:26.179790 32097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:26.180194 32097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:26.185209 32097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "0285ac6bc0674a2aa1dbebecea2b245f"
format_stamp: "Formatted at 2024-05-02 14:05:26 on dist-test-slave-dlb8"
server_key: "b39335d466bc9f6284740b0a99b4ddf1"
server_key_iv: "1e5d8d6db29785f3af7fbed13ca7a25e"
server_key_version: "encryptionkey@0"
I20240502 14:05:26.186275 32097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "0285ac6bc0674a2aa1dbebecea2b245f"
format_stamp: "Formatted at 2024-05-02 14:05:26 on dist-test-slave-dlb8"
server_key: "b39335d466bc9f6284740b0a99b4ddf1"
server_key_iv: "1e5d8d6db29785f3af7fbed13ca7a25e"
server_key_version: "encryptionkey@0"
I20240502 14:05:26.193322 32097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 14:05:26.198860 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:26.199849 32097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:05:26.200134 32097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "0285ac6bc0674a2aa1dbebecea2b245f"
format_stamp: "Formatted at 2024-05-02 14:05:26 on dist-test-slave-dlb8"
server_key: "b39335d466bc9f6284740b0a99b4ddf1"
server_key_iv: "1e5d8d6db29785f3af7fbed13ca7a25e"
server_key_version: "encryptionkey@0"
I20240502 14:05:26.200443 32097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:26.253630 32097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:26.258870 32097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:26.259279 32097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:26.262151 32097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:26.265789 32097 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:26.265957 32097 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:26.266269 32097 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:26.266415 32097 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:26.385843 32097 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38731
I20240502 14:05:26.385939 32226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38731 every 8 connection(s)
I20240502 14:05:26.388226 32097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:05:26.396412 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32097
I20240502 14:05:26.396956 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:05:26.398375 28345 external_mini_cluster.cc:1428] Setting key 99b91ffe4c96b548ae5e2120b39ef7db
May 02 14:05:26 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658726, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:05:26.426970 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:26.392158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43897 (local address 127.27.174.126:46801)
0502 14:05:26.393218 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:26.393234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:26.395490 (+ 2256us) server_negotiation.cc:407] Connection header received
0502 14:05:26.401569 (+ 6079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:26.401599 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:26.401834 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:26.402023 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:26.405699 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:26.407494 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:26.409420 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:26.409773 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:26.414870 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:26.414984 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:26.415022 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:26.415196 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:26.420820 (+ 5624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:26.422111 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:26.422151 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:26.422183 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:26.422393 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:26.423066 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:26.423101 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:26.423133 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:26.423635 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:26.423810 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:26.424790 (+ 980us) server_negotiation.cc:299] Negotiation successful
0502 14:05:26.425135 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":427,"threads_started":1}
I20240502 14:05:26.427529 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:26.392502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46801 (local address 127.27.174.65:43897)
0502 14:05:26.395272 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:26.395354 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:26.401187 (+ 5833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:26.402331 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:26.402376 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:26.403559 (+ 1183us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:26.404890 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:26.405030 (+ 140us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:26.407752 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:26.407790 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:26.409141 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:26.409203 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:26.409693 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:26.410713 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:26.410818 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:26.414515 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:26.421080 (+ 6565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:26.421105 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:26.421132 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:26.421852 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:26.422589 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:26.422608 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:26.422623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:26.422843 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:26.423820 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:26.423842 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:26.424270 (+ 428us) client_negotiation.cc:769] Sending connection context
0502 14:05:26.424616 (+ 346us) client_negotiation.cc:240] Negotiation successful
0502 14:05:26.425083 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2110,"thread_start_us":1575,"threads_started":1}
I20240502 14:05:26.429603 32227 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46801
I20240502 14:05:26.430124 32227 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:26.431519 32227 heartbeater.cc:511] Master 127.27.174.126:46801 requested a full tablet report, sending...
I20240502 14:05:26.434752 32036 ts_manager.cc:194] Registered new tserver with Master: 0285ac6bc0674a2aa1dbebecea2b245f (127.27.174.65:38731)
I20240502 14:05:26.437467 32036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:43897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:05:26.454578 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:26.747359 32233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:26.747869 32233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:26.748499 32233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:26.777086 32233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:26.777467 32233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:26.778416 32233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:05:26.811107 32233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:26.812242 32233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:26.813212 32233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:26.828564 32239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:27.441237 32227 heartbeater.cc:503] Master 127.27.174.126:46801 was elected leader, sending a full tablet report...
W20240502 14:05:26.829396 32242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:26.830760 32233 server_base.cc:1034] running on GCE node
W20240502 14:05:26.830258 32240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:27.867630 32233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:27.870399 32233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:27.871814 32233 hybrid_clock.cc:648] HybridClock initialized: now 1714658727871781 us; error 54 us; skew 500 ppm
I20240502 14:05:27.872565 32233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:27 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658727, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:27.878083 32233 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:05:27.883075 32233 webserver.cc:460] Webserver started at http://127.27.174.66:38067/ using document root <none> and password file <none>
I20240502 14:05:27.883915 32233 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:27.884114 32233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:27.884526 32233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:27.889122 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "300277d53b9145bda65dc8761c1543d8"
format_stamp: "Formatted at 2024-05-02 14:05:27 on dist-test-slave-dlb8"
server_key: "fcfab7336d9afe5839b43dd525549288"
server_key_iv: "bc9a7a99ecdefdb51f74d918598aa4c1"
server_key_version: "encryptionkey@0"
I20240502 14:05:27.890199 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "300277d53b9145bda65dc8761c1543d8"
format_stamp: "Formatted at 2024-05-02 14:05:27 on dist-test-slave-dlb8"
server_key: "fcfab7336d9afe5839b43dd525549288"
server_key_iv: "bc9a7a99ecdefdb51f74d918598aa4c1"
server_key_version: "encryptionkey@0"
I20240502 14:05:27.896916 32233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:27.902284 32251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:27.903257 32233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:05:27.903553 32233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "300277d53b9145bda65dc8761c1543d8"
format_stamp: "Formatted at 2024-05-02 14:05:27 on dist-test-slave-dlb8"
server_key: "fcfab7336d9afe5839b43dd525549288"
server_key_iv: "bc9a7a99ecdefdb51f74d918598aa4c1"
server_key_version: "encryptionkey@0"
I20240502 14:05:27.903884 32233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:27.958571 32233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:27.963377 32233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:27.963798 32233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:27.966061 32233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:27.969628 32233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:27.969813 32233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:27.970060 32233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:27.970208 32233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 14:05:28.091034 32233 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:33917
I20240502 14:05:28.091117 32363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:33917 every 8 connection(s)
I20240502 14:05:28.093472 32233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:05:28.099470 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32233
I20240502 14:05:28.100077 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:05:28.101111 28345 external_mini_cluster.cc:1428] Setting key d6d09d1947b0d472139e17ff0f7eb8a2
May 02 14:05:28 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658727, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:05:28.125252 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:28.097259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48837 (local address 127.27.174.126:46801)
0502 14:05:28.098322 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:28.098342 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:28.100677 (+ 2335us) server_negotiation.cc:407] Connection header received
0502 14:05:28.106025 (+ 5348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:28.106079 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:28.106231 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:28.106366 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:28.108548 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:28.109333 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:28.110653 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:28.111179 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:28.115517 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:28.115588 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:28.115604 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:28.115696 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:28.119419 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:28.120861 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:28.120895 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:28.120908 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:28.121030 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:28.121709 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:28.121730 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:28.121753 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:28.122165 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:28.122395 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:28.123303 (+ 908us) server_negotiation.cc:299] Negotiation successful
0502 14:05:28.123686 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":366,"threads_started":1}
I20240502 14:05:28.126099 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:28.097508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46801 (local address 127.27.174.66:48837)
0502 14:05:28.100462 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:28.100525 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:28.105703 (+ 5178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:28.106637 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:28.106662 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:28.107396 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:28.108301 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:28.108338 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:28.109560 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:28.109579 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:28.110425 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:28.110463 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:28.110819 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:28.111872 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:28.111938 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:28.115214 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:28.119630 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:28.119667 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:28.119706 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:28.120665 (+ 959us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:28.121201 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:28.121233 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:28.121264 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:28.121538 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:28.122450 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:28.122478 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:28.122851 (+ 373us) client_negotiation.cc:769] Sending connection context
0502 14:05:28.123163 (+ 312us) client_negotiation.cc:240] Negotiation successful
0502 14:05:28.123629 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2266,"thread_start_us":1689,"threads_started":1}
I20240502 14:05:28.128144 32364 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46801
I20240502 14:05:28.128580 32364 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:28.129730 32364 heartbeater.cc:511] Master 127.27.174.126:46801 requested a full tablet report, sending...
I20240502 14:05:28.132118 32036 ts_manager.cc:194] Registered new tserver with Master: 300277d53b9145bda65dc8761c1543d8 (127.27.174.66:33917)
I20240502 14:05:28.133937 32036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:48837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:05:28.150203 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:28.439287 32370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:28.439747 32370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:28.440387 32370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:28.469695 32370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:28.469995 32370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:28.470914 32370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:05:28.502805 32370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:28.504240 32370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:28.505537 32370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:28.525661 32379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:29.136934 32364 heartbeater.cc:503] Master 127.27.174.126:46801 was elected leader, sending a full tablet report...
W20240502 14:05:28.525704 32376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:28.527173 32370 server_base.cc:1034] running on GCE node
W20240502 14:05:28.526597 32377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:29.568014 32370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:29.570099 32370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:29.571411 32370 hybrid_clock.cc:648] HybridClock initialized: now 1714658729571393 us; error 45 us; skew 500 ppm
I20240502 14:05:29.572166 32370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:29 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:29.582616 32370 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:05:29.586745 32370 webserver.cc:460] Webserver started at http://127.27.174.67:45463/ using document root <none> and password file <none>
I20240502 14:05:29.587563 32370 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:29.587769 32370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:29.588168 32370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:29.592902 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "e4079e3a82244fc5a231ae250fb57540"
format_stamp: "Formatted at 2024-05-02 14:05:29 on dist-test-slave-dlb8"
server_key: "043b3731cbf5c6bd0c72685359dd0f34"
server_key_iv: "da67acb92842327f33223dbb11c985d5"
server_key_version: "encryptionkey@0"
I20240502 14:05:29.593956 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "e4079e3a82244fc5a231ae250fb57540"
format_stamp: "Formatted at 2024-05-02 14:05:29 on dist-test-slave-dlb8"
server_key: "043b3731cbf5c6bd0c72685359dd0f34"
server_key_iv: "da67acb92842327f33223dbb11c985d5"
server_key_version: "encryptionkey@0"
I20240502 14:05:29.600605 32370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:05:29.605824 32387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:29.606738 32370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:05:29.607019 32370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "e4079e3a82244fc5a231ae250fb57540"
format_stamp: "Formatted at 2024-05-02 14:05:29 on dist-test-slave-dlb8"
server_key: "043b3731cbf5c6bd0c72685359dd0f34"
server_key_iv: "da67acb92842327f33223dbb11c985d5"
server_key_version: "encryptionkey@0"
I20240502 14:05:29.607357 32370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:29.660656 32370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:29.666079 32370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:29.666530 32370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:29.668889 32370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:29.672726 32370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:29.672947 32370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:29.673202 32370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:29.673347 32370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:29.798002 32370 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:39499
I20240502 14:05:29.798182 32499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:39499 every 8 connection(s)
I20240502 14:05:29.800442 32370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:05:29.804266 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32370
I20240502 14:05:29.804836 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:05:29.806524 28345 external_mini_cluster.cc:1428] Setting key 2e111d1be1dfec972658427973f7251e
May 02 14:05:29 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658729, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:29.830266 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:29.804808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:53419 (local address 127.27.174.126:46801)
0502 14:05:29.806148 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:29.806162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:29.807461 (+ 1299us) server_negotiation.cc:407] Connection header received
0502 14:05:29.812113 (+ 4652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:29.812133 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:29.812304 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:29.812441 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:29.814589 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.815772 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:29.816964 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.817289 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:29.821004 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:29.821079 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:29.821095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:29.821192 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:29.825468 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.826537 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.826561 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.826575 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.826711 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.827269 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.827287 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.827300 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.827681 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:29.827872 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:29.828653 (+ 781us) server_negotiation.cc:299] Negotiation successful
0502 14:05:29.828953 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":657,"threads_started":1}
I20240502 14:05:29.830694 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:29.804829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46801 (local address 127.27.174.67:53419)
0502 14:05:29.807223 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:29.807299 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:29.811815 (+ 4516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:29.812643 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:29.812671 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:29.813439 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:29.814385 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:29.814424 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.816003 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:29.816025 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:29.816784 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:29.816818 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.817081 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:29.817858 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:29.817919 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:29.820749 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:29.825653 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:29.825674 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:29.825695 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:29.826372 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:29.826856 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:29.826876 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:29.826890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:29.827076 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:29.827833 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:29.827853 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:29.828224 (+ 371us) client_negotiation.cc:769] Sending connection context
0502 14:05:29.828496 (+ 272us) client_negotiation.cc:240] Negotiation successful
0502 14:05:29.828881 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1887,"thread_start_us":1433,"threads_started":1}
I20240502 14:05:29.832463 32500 heartbeater.cc:347] Connected to a master server at 127.27.174.126:46801
I20240502 14:05:29.832839 32500 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:29.833781 32500 heartbeater.cc:511] Master 127.27.174.126:46801 requested a full tablet report, sending...
I20240502 14:05:29.836010 32036 ts_manager.cc:194] Registered new tserver with Master: e4079e3a82244fc5a231ae250fb57540 (127.27.174.67:39499)
I20240502 14:05:29.837483 32036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:53419
I20240502 14:05:29.839246 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:05:29 dist-test-slave-dlb8 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 02 14:05:29 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:05:29.900362 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:29.879984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.27.174.65:38731)
0502 14:05:29.881665 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:29.881684 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:29.881881 (+ 197us) server_negotiation.cc:407] Connection header received
0502 14:05:29.882192 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:29.882218 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:29.882459 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:29.882623 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:29.884981 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.886366 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:29.887604 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.887935 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:29.892092 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:29.892197 (+ 105us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:29.892234 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:29.892328 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:29.895696 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.896711 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.896730 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.896756 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.896885 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.897283 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.897298 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.897309 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.897752 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:29.898103 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:29.898517 (+ 414us) server_negotiation.cc:299] Negotiation successful
0502 14:05:29.898886 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":1045,"threads_started":1}
May 02 14:05:29 dist-test-slave-dlb8 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:29.923247 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:29.906732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.27.174.126:46801)
0502 14:05:29.907435 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:29.907452 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:29.907680 (+ 228us) server_negotiation.cc:407] Connection header received
0502 14:05:29.907925 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:29.907941 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:29.908093 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:29.908210 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:29.909912 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.910710 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:29.911786 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:29.912136 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:29.915880 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:29.915948 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:29.915966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:29.916052 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:29.919040 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.919845 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.919863 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.919876 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.919999 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:29.920430 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:29.920445 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:29.920457 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:29.920818 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:29.921055 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:29.921457 (+ 402us) server_negotiation.cc:299] Negotiation successful
0502 14:05:29.921784 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20240502 14:05:29.924151 32035 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59404
I20240502 14:05:29.925560 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32097
I20240502 14:05:29.946774 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32233
I20240502 14:05:29.968755 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32370
I20240502 14:05:29.992368 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32002
2024-05-02T14:05:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (8217 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:30 dist-test-slave-dlb8 krb5kdc[32513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:30 dist-test-slave-dlb8 krb5kdc[32513](info): set up 2 sockets
May 02 14:05:30 dist-test-slave-dlb8 krb5kdc[32513](info): commencing operation
krb5kdc: starting...
W20240502 14:05:30.955796 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.909s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:31 dist-test-slave-dlb8 krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:31.154129 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34719
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:31.433884 32526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:31.434450 32526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:31.435000 32526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:31.463687 32526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:31.463994 32526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:31.464249 32526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:31.464532 32526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:31.464745 32526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:31.498013 32526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:31.499037 32526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:31.499935 32526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:31.515370 32532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:31.516148 32533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:31.516975 32526 server_base.cc:1034] running on GCE node
W20240502 14:05:31.517060 32535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:32.578557 32526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:32.581007 32526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:32.582329 32526 hybrid_clock.cc:648] HybridClock initialized: now 1714658732582292 us; error 54 us; skew 500 ppm
I20240502 14:05:32.583122 32526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:32 dist-test-slave-dlb8 krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658732, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:32.588719 32526 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:05:32.591969 32526 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:32.592234 32526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:32.592670 32526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:32.597553 32526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90"
format_stamp: "Formatted at 2024-05-02 14:05:32 on dist-test-slave-dlb8"
server_key: "716045dbc818a4a06f3be3589bfafaff"
server_key_iv: "91f0ad32c01bf45e33be520a289b3a22"
server_key_version: "encryptionkey@0"
I20240502 14:05:32.598678 32526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90"
format_stamp: "Formatted at 2024-05-02 14:05:32 on dist-test-slave-dlb8"
server_key: "716045dbc818a4a06f3be3589bfafaff"
server_key_iv: "91f0ad32c01bf45e33be520a289b3a22"
server_key_version: "encryptionkey@0"
I20240502 14:05:32.605422 32526 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:32.610608 32542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:32.611594 32526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:05:32.611908 32526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90"
format_stamp: "Formatted at 2024-05-02 14:05:32 on dist-test-slave-dlb8"
server_key: "716045dbc818a4a06f3be3589bfafaff"
server_key_iv: "91f0ad32c01bf45e33be520a289b3a22"
server_key_version: "encryptionkey@0"
I20240502 14:05:32.612306 32526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:32.658142 32526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:32.663419 32526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:32.663847 32526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:32.733026 32526 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34719
I20240502 14:05:32.733074 32593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34719 every 8 connection(s)
I20240502 14:05:32.734230 32526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:32.739132 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:32.742444 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32526
I20240502 14:05:32.743031 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:32.744580 28345 external_mini_cluster.cc:1428] Setting key 5b4a6ff1e2328e8a4511c972b1d0d0d5
I20240502 14:05:32.762369 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: Bootstrap starting.
May 02 14:05:32 dist-test-slave-dlb8 krb5kdc[32513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:32.767611 32594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:32.769282 32594 log.cc:824] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:32.774636 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: No bootstrap required, opened a new log
I20240502 14:05:32.777078 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:32.747690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.27.174.126:34719)
0502 14:05:32.750512 (+ 2822us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:32.750539 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:32.750592 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:05:32.754613 (+ 4021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:32.754656 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:32.754989 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:32.755310 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:32.758554 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:32.759928 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:32.761677 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:32.762170 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:32.766882 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:32.766978 (+ 96us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:32.767007 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:32.767112 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:32.771133 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:32.772192 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:32.772218 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:32.772250 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:32.772437 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:32.772992 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:32.773012 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:32.773025 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:32.773611 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:32.774060 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:32.774706 (+ 646us) server_negotiation.cc:299] Negotiation successful
0502 14:05:32.775277 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":1788,"threads_started":1}
I20240502 14:05:32.794924 32594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } }
I20240502 14:05:32.795722 32594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:32.795929 32594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c08ef2dd9a4ee7a96cf9ba489afc90, State: Initialized, Role: FOLLOWER
I20240502 14:05:32.796604 32594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } }
I20240502 14:05:32.797070 32594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:32.797312 32594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:32.797617 32594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:32.801327 32594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } }
I20240502 14:05:32.801978 32594 leader_election.cc:302] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c08ef2dd9a4ee7a96cf9ba489afc90; no voters:
I20240502 14:05:32.803659 32594 leader_election.cc:288] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:32.804065 32599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:32.806166 32599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [term 1 LEADER]: Becoming Leader. State: Replica: 41c08ef2dd9a4ee7a96cf9ba489afc90, State: Running, Role: LEADER
I20240502 14:05:32.806993 32599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } }
I20240502 14:05:32.808084 32594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:32.816890 32600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c08ef2dd9a4ee7a96cf9ba489afc90. Latest consensus state: current_term: 1 leader_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } } }
I20240502 14:05:32.817037 32601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c08ef2dd9a4ee7a96cf9ba489afc90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34719 } } }
I20240502 14:05:32.818104 32601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:32.818151 32600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:32.828372 32608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:32.840121 32608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:32.857642 32608 catalog_manager.cc:1347] Generated new cluster ID: b768d12991104054a4ae522d58b90637
I20240502 14:05:32.857887 32608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:32.877501 32608 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:32.879185 32608 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:32.898594 32608 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: Generated new TSK 0
I20240502 14:05:32.899416 32608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:05:32.955675 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:33.239758 32620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:33.240244 32620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:33.240835 32620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:33.269770 32620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:33.270128 32620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:33.270980 32620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:05:33.304385 32620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:33.305474 32620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:33.306430 32620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:33.324769 32627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:33.326501 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:33.326239 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:33.327494 32620 server_base.cc:1034] running on GCE node
I20240502 14:05:34.408854 32620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:34.411808 32620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:34.413275 32620 hybrid_clock.cc:648] HybridClock initialized: now 1714658734413215 us; error 80 us; skew 500 ppm
I20240502 14:05:34.414108 32620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:34 dist-test-slave-dlb8 krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658734, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:34.420460 32620 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:05:34.424299 32620 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.424499 32620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.424928 32620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.429900 32620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "9a6739928498426a9094f8fb136eb217"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-dlb8"
server_key: "78973f8b81b2786ddde94c6a8822de90"
server_key_iv: "c1ef79f8858883ef89a1965f11f0be3e"
server_key_version: "encryptionkey@0"
I20240502 14:05:34.431047 32620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "9a6739928498426a9094f8fb136eb217"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-dlb8"
server_key: "78973f8b81b2786ddde94c6a8822de90"
server_key_iv: "c1ef79f8858883ef89a1965f11f0be3e"
server_key_version: "encryptionkey@0"
I20240502 14:05:34.438616 32620 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240502 14:05:34.444659 32636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.445849 32620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:05:34.446218 32620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "9a6739928498426a9094f8fb136eb217"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-dlb8"
server_key: "78973f8b81b2786ddde94c6a8822de90"
server_key_iv: "c1ef79f8858883ef89a1965f11f0be3e"
server_key_version: "encryptionkey@0"
I20240502 14:05:34.446588 32620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.493748 32620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.498673 32620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:34.499073 32620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.501449 32620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:34.505290 32620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:34.505473 32620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.505710 32620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:34.505843 32620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.632241 32620 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44837
I20240502 14:05:34.632341 32748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44837 every 8 connection(s)
I20240502 14:05:34.633316 32620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:05:34.642091 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32620
I20240502 14:05:34.642725 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:05:34.644255 28345 external_mini_cluster.cc:1428] Setting key 52bd15a1ab985247f7c36640a208f4ba
May 02 14:05:34 dist-test-slave-dlb8 krb5kdc[32513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658734, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:05:34.674926 32751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:34.636749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57707 (local address 127.27.174.126:34719)
0502 14:05:34.637993 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:34.638011 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:34.640339 (+ 2328us) server_negotiation.cc:407] Connection header received
0502 14:05:34.646603 (+ 6264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:34.646623 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:34.646786 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:34.646934 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:34.649831 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:34.650717 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:34.652673 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:34.653035 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:34.664710 (+ 11675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:34.664787 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:34.664804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:34.664905 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:34.668659 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:34.670223 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:34.670250 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:34.670267 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:34.670427 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:34.671208 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:34.671228 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:34.671241 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:34.671615 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:34.671780 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:34.673034 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0502 14:05:34.673367 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":563,"threads_started":1}
I20240502 14:05:34.676219 32752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:34.637066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34719 (local address 127.27.174.65:57707)
0502 14:05:34.640087 (+ 3021us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:34.640153 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:34.646238 (+ 6085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:34.647232 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:34.647259 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:34.648206 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:34.649531 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:34.649593 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:34.650971 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:34.651013 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:34.652389 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:34.652448 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:34.652879 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:34.660757 (+ 7878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:34.660827 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:34.664350 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:34.668902 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:34.668946 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:34.668989 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:34.670002 (+ 1013us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:34.670609 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:34.670648 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:34.670679 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:34.670984 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:34.671821 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:34.671858 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:34.672429 (+ 571us) client_negotiation.cc:769] Sending connection context
0502 14:05:34.672888 (+ 459us) client_negotiation.cc:240] Negotiation successful
0502 14:05:34.673569 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2337,"thread_start_us":1712,"threads_started":1}
I20240502 14:05:34.678196 32749 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34719
I20240502 14:05:34.678594 32749 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:34.679478 32749 heartbeater.cc:511] Master 127.27.174.126:34719 requested a full tablet report, sending...
I20240502 14:05:34.682197 32559 ts_manager.cc:194] Registered new tserver with Master: 9a6739928498426a9094f8fb136eb217 (127.27.174.65:44837)
I20240502 14:05:34.684954 32559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:57707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:05:34.696489 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:35.003482 32755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:35.003942 32755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:35.004611 32755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:35.034265 32755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:35.034611 32755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:35.035494 32755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:05:35.068397 32755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:35.069540 32755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:35.070556 32755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:35.085369 32762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:35.689088 32749 heartbeater.cc:503] Master 127.27.174.126:34719 was elected leader, sending a full tablet report...
W20240502 14:05:35.087023 32764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:35.087740 32755 server_base.cc:1034] running on GCE node
W20240502 14:05:35.086385 32761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:36.119635 32755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:36.122462 32755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:36.123894 32755 hybrid_clock.cc:648] HybridClock initialized: now 1714658736123857 us; error 60 us; skew 500 ppm
I20240502 14:05:36.124650 32755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:36 dist-test-slave-dlb8 krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658736, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:36.130712 32755 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:05:36.134052 32755 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:36.134300 32755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:36.134737 32755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:36.139663 32755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "7e7acdfe1fb0461ebf19cbe02b040f99"
format_stamp: "Formatted at 2024-05-02 14:05:36 on dist-test-slave-dlb8"
server_key: "142e88ea7d2389599aa9f516e9069178"
server_key_iv: "4882c3045f1c3eb7e6d5d271892a86ab"
server_key_version: "encryptionkey@0"
I20240502 14:05:36.140767 32755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "7e7acdfe1fb0461ebf19cbe02b040f99"
format_stamp: "Formatted at 2024-05-02 14:05:36 on dist-test-slave-dlb8"
server_key: "142e88ea7d2389599aa9f516e9069178"
server_key_iv: "4882c3045f1c3eb7e6d5d271892a86ab"
server_key_version: "encryptionkey@0"
I20240502 14:05:36.147671 32755 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20240502 14:05:36.153070 303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:36.154160 32755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:05:36.154474 32755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "7e7acdfe1fb0461ebf19cbe02b040f99"
format_stamp: "Formatted at 2024-05-02 14:05:36 on dist-test-slave-dlb8"
server_key: "142e88ea7d2389599aa9f516e9069178"
server_key_iv: "4882c3045f1c3eb7e6d5d271892a86ab"
server_key_version: "encryptionkey@0"
I20240502 14:05:36.154832 32755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:36.219089 32755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:36.224473 32755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:36.224901 32755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:36.227334 32755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:36.231096 32755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:36.231304 32755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:36.231561 32755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:36.231707 32755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:36.358213 32755 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:39543
I20240502 14:05:36.358256 415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:39543 every 8 connection(s)
I20240502 14:05:36.359225 32755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:05:36.362560 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 32755
I20240502 14:05:36.363090 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:05:36.364627 28345 external_mini_cluster.cc:1428] Setting key 3e04a2c05709a373b083df3cc32cbb52
May 02 14:05:36 dist-test-slave-dlb8 krb5kdc[32513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658736, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:05:36.392014 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:36.362972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34719 (local address 127.27.174.66:40073)
0502 14:05:36.365659 (+ 2687us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:36.365724 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:36.370964 (+ 5240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:36.372019 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:36.372045 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:36.372880 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:36.373836 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:36.373875 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:36.375224 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:36.375247 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:36.376315 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:36.376373 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:36.376814 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:36.377757 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:36.377858 (+ 101us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:36.381767 (+ 3909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:36.385965 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:36.385998 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:36.386030 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:36.386887 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:36.387535 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:36.387591 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:36.387613 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:36.387822 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:36.388573 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:36.388592 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:36.389031 (+ 439us) client_negotiation.cc:769] Sending connection context
0502 14:05:36.389549 (+ 518us) client_negotiation.cc:240] Negotiation successful
0502 14:05:36.390022 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"thread_start_us":1635,"threads_started":1}
I20240502 14:05:36.392611 418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:36.363017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:40073 (local address 127.27.174.126:34719)
0502 14:05:36.364562 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:36.364579 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:36.365892 (+ 1313us) server_negotiation.cc:407] Connection header received
0502 14:05:36.371356 (+ 5464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:36.371382 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:36.371553 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:36.371721 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:36.374116 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:36.374984 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:36.376616 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:36.376973 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:36.382094 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:36.382169 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:36.382186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:36.382289 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:36.385745 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:36.387062 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:36.387087 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:36.387099 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:36.387228 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:36.387987 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:36.388007 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:36.388019 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:36.388382 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:36.388547 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:36.389731 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0502 14:05:36.390259 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":893,"threads_started":1}
I20240502 14:05:36.394308 416 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34719
I20240502 14:05:36.394791 416 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:36.396075 416 heartbeater.cc:511] Master 127.27.174.126:34719 requested a full tablet report, sending...
I20240502 14:05:36.398674 32559 ts_manager.cc:194] Registered new tserver with Master: 7e7acdfe1fb0461ebf19cbe02b040f99 (127.27.174.66:39543)
I20240502 14:05:36.400475 32559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:40073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:05:36.418249 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:36.710713 422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:36.711165 422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:36.711757 422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:36.752488 422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:36.752962 422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:36.754542 422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:05:36.791018 422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:36.792089 422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:36.793051 422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:36.808833 431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:37.403852 416 heartbeater.cc:503] Master 127.27.174.126:34719 was elected leader, sending a full tablet report...
W20240502 14:05:36.809739 429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:36.810767 422 server_base.cc:1034] running on GCE node
W20240502 14:05:36.809980 428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:37.850732 422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:37.852903 422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:37.854247 422 hybrid_clock.cc:648] HybridClock initialized: now 1714658737854209 us; error 62 us; skew 500 ppm
I20240502 14:05:37.854961 422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:05:37 dist-test-slave-dlb8 krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658737, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:05:37.860462 422 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:05:37.863233 422 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:37.863456 422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:37.863903 422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:37.868575 422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "011a57ffa47841b7af03dc586c08874a"
format_stamp: "Formatted at 2024-05-02 14:05:37 on dist-test-slave-dlb8"
server_key: "445b700af5b41fd20d5e810153811829"
server_key_iv: "704ef25f3b17a9c12d93db1fa8a3d637"
server_key_version: "encryptionkey@0"
I20240502 14:05:37.869562 422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "011a57ffa47841b7af03dc586c08874a"
format_stamp: "Formatted at 2024-05-02 14:05:37 on dist-test-slave-dlb8"
server_key: "445b700af5b41fd20d5e810153811829"
server_key_iv: "704ef25f3b17a9c12d93db1fa8a3d637"
server_key_version: "encryptionkey@0"
I20240502 14:05:37.876117 422 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:05:37.881438 439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:37.882534 422 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:05:37.882840 422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "011a57ffa47841b7af03dc586c08874a"
format_stamp: "Formatted at 2024-05-02 14:05:37 on dist-test-slave-dlb8"
server_key: "445b700af5b41fd20d5e810153811829"
server_key_iv: "704ef25f3b17a9c12d93db1fa8a3d637"
server_key_version: "encryptionkey@0"
I20240502 14:05:37.883188 422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:37.935329 422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:37.940249 422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:37.940665 422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:37.943104 422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:37.946952 422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:37.947153 422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:37.947372 422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:37.947520 422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:38.068380 422 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:44393
I20240502 14:05:38.068473 551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:44393 every 8 connection(s)
I20240502 14:05:38.069366 422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:05:38.074357 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 422
I20240502 14:05:38.074954 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:05:38.076002 28345 external_mini_cluster.cc:1428] Setting key 6e715a20df9e35f82774ab2b79ab3203
May 02 14:05:38 dist-test-slave-dlb8 krb5kdc[32513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658737, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:05:38.099234 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.072568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:58927 (local address 127.27.174.126:34719)
0502 14:05:38.073684 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.073708 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.075904 (+ 2196us) server_negotiation.cc:407] Connection header received
0502 14:05:38.080666 (+ 4762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.080708 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.080862 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.080988 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:38.083249 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.084158 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.085281 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.085590 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.089289 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:38.089360 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:38.089375 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:38.089464 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:38.093187 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:38.094560 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:38.094584 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:38.094597 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:38.094726 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:38.095297 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:38.095328 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:38.095340 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:38.095686 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:38.095887 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.096742 (+ 855us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.097145 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":409,"threads_started":1}
I20240502 14:05:38.099813 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.072888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34719 (local address 127.27.174.67:58927)
0502 14:05:38.075684 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.075745 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.080334 (+ 4589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.081275 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.081305 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.082094 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:38.083020 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.083064 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.084354 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.084375 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.085095 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.085144 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.085406 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.086195 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:05:38.086260 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:38.089045 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:38.093418 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:38.093442 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:38.093485 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:38.094372 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:38.094892 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:05:38.094910 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:05:38.094924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:05:38.095114 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:05:38.095862 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:38.095890 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:38.096248 (+ 358us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.096547 (+ 299us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.097099 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2140,"thread_start_us":1608,"threads_started":1}
I20240502 14:05:38.101331 552 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34719
I20240502 14:05:38.101658 552 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:38.102618 552 heartbeater.cc:511] Master 127.27.174.126:34719 requested a full tablet report, sending...
I20240502 14:05:38.104558 32559 ts_manager.cc:194] Registered new tserver with Master: 011a57ffa47841b7af03dc586c08874a (127.27.174.67:44393)
I20240502 14:05:38.105674 32559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:58927
I20240502 14:05:38.108227 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:05:38.139837 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.122954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.27.174.126:34719)
0502 14:05:38.123824 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.123848 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.124485 (+ 637us) server_negotiation.cc:407] Connection header received
0502 14:05:38.124801 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.124817 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.124975 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.125100 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:38.127602 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.128393 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.129585 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.129895 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.132259 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:38.132325 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:38.132341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:05:38.132424 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:38.135461 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:38.136406 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:38.136423 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:38.136435 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:38.136556 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:05:38.137067 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:05:38.137081 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:05:38.137111 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:05:38.137493 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:38.137728 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.138224 (+ 496us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.138539 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:05:38.145390 32559 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:05:38.164351 32559 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:05:38.215207 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.190782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:39543 (local address 127.0.0.1:56912)
0502 14:05:38.205366 (+ 14584us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.205434 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.205687 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.207855 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.207892 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.208027 (+ 135us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:05:38.208945 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.208996 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.211170 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.211197 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.212694 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.212745 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.213051 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.213191 (+ 140us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.213380 (+ 189us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.213615 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14184,"thread_start_us":11781,"threads_started":1}
I20240502 14:05:38.216845 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.189737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44837 (local address 127.0.0.1:58022)
0502 14:05:38.204847 (+ 15110us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.205090 (+ 243us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.205342 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.208532 (+ 3190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.208564 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.208687 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:05:38.209596 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.209644 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.212653 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.212685 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.214611 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.214677 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.214981 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.215065 (+ 84us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.215189 (+ 124us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.215392 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14333,"thread_start_us":13850,"threads_started":1}
I20240502 14:05:38.217646 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.192665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.27.174.66:39543)
0502 14:05:38.204747 (+ 12082us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.204768 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.205583 (+ 815us) server_negotiation.cc:407] Connection header received
0502 14:05:38.205973 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.206021 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.206905 (+ 884us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.207237 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:05:38.209226 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.210881 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.213177 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.214654 (+ 1477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.214901 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.215100 (+ 199us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.215323 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11654,"thread_start_us":11245,"threads_started":1}
I20240502 14:05:38.220088 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.188536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44393 (local address 127.0.0.1:40514)
0502 14:05:38.201839 (+ 13303us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.201946 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.202356 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.210189 (+ 7833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.210215 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.210307 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:05:38.211146 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.211190 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.215735 (+ 4545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.215765 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.217614 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.217671 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.217977 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.218087 (+ 110us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.218217 (+ 130us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.218421 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12421,"thread_start_us":16196,"threads_started":1}
I20240502 14:05:38.221812 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.189019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.27.174.67:44393)
0502 14:05:38.209318 (+ 20299us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.209353 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.209404 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:05:38.209573 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.209601 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.209903 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.210138 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:05:38.213210 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.215464 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.218751 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.220087 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.220331 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.220549 (+ 218us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.220780 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19498,"thread_start_us":19106,"threads_started":1}
I20240502 14:05:38.222321 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.191435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.27.174.65:44837)
0502 14:05:38.207577 (+ 16142us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.207611 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.207661 (+ 50us) server_negotiation.cc:407] Connection header received
0502 14:05:38.207825 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.207856 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.208236 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.209863 (+ 1627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:05:38.210310 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.212370 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.217326 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.220302 (+ 2976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.220534 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.220740 (+ 206us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.220989 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15539,"thread_start_us":14304,"threads_started":1}
I20240502 14:05:38.222290 351 tablet_service.cc:1467] Processing CreateTablet for tablet f5c4259477db4245b968d771d3b5975c (DEFAULT_TABLE table=test-table [id=c5e3da1a1d554611865140bd5c681015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:38.225260 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c4259477db4245b968d771d3b5975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:38.225945 32684 tablet_service.cc:1467] Processing CreateTablet for tablet f5c4259477db4245b968d771d3b5975c (DEFAULT_TABLE table=test-table [id=c5e3da1a1d554611865140bd5c681015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:38.226238 487 tablet_service.cc:1467] Processing CreateTablet for tablet f5c4259477db4245b968d771d3b5975c (DEFAULT_TABLE table=test-table [id=c5e3da1a1d554611865140bd5c681015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:38.227303 32684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c4259477db4245b968d771d3b5975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:38.227979 487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c4259477db4245b968d771d3b5975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:38.247162 571 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Bootstrap starting.
I20240502 14:05:38.251850 571 tablet_bootstrap.cc:654] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:38.254170 572 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Bootstrap starting.
I20240502 14:05:38.255621 573 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Bootstrap starting.
I20240502 14:05:38.257293 571 log.cc:824] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:38.260995 573 tablet_bootstrap.cc:654] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:38.261274 572 tablet_bootstrap.cc:654] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:38.262727 573 log.cc:824] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:38.263442 571 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: No bootstrap required, opened a new log
I20240502 14:05:38.263479 572 log.cc:824] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:38.263975 571 ts_tablet_manager.cc:1367] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.002s
I20240502 14:05:38.268469 573 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: No bootstrap required, opened a new log
I20240502 14:05:38.268846 573 ts_tablet_manager.cc:1367] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20240502 14:05:38.269614 572 tablet_bootstrap.cc:492] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: No bootstrap required, opened a new log
I20240502 14:05:38.270105 572 ts_tablet_manager.cc:1367] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240502 14:05:38.290218 571 raft_consensus.cc:348] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.290870 571 raft_consensus.cc:374] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:38.291111 571 raft_consensus.cc:729] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6739928498426a9094f8fb136eb217, State: Initialized, Role: FOLLOWER
I20240502 14:05:38.291862 571 consensus_queue.cc:260] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.291923 573 raft_consensus.cc:348] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.292835 573 raft_consensus.cc:374] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:38.293215 573 raft_consensus.cc:729] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011a57ffa47841b7af03dc586c08874a, State: Initialized, Role: FOLLOWER
I20240502 14:05:38.294219 573 consensus_queue.cc:260] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.295755 571 ts_tablet_manager.cc:1398] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Time spent starting tablet: real 0.031s user 0.025s sys 0.003s
I20240502 14:05:38.297156 552 heartbeater.cc:503] Master 127.27.174.126:34719 was elected leader, sending a full tablet report...
I20240502 14:05:38.300873 573 ts_tablet_manager.cc:1398] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Time spent starting tablet: real 0.032s user 0.027s sys 0.003s
I20240502 14:05:38.302881 572 raft_consensus.cc:348] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.303814 572 raft_consensus.cc:374] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:38.304108 572 raft_consensus.cc:729] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7acdfe1fb0461ebf19cbe02b040f99, State: Initialized, Role: FOLLOWER
I20240502 14:05:38.304929 572 consensus_queue.cc:260] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.308547 572 ts_tablet_manager.cc:1398] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
W20240502 14:05:38.323101 553 tablet.cc:2369] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:38.346483 577 raft_consensus.cc:482] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:38.346906 577 raft_consensus.cc:504] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.349191 577 leader_election.cc:288] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011a57ffa47841b7af03dc586c08874a (127.27.174.67:44393), 7e7acdfe1fb0461ebf19cbe02b040f99 (127.27.174.66:39543)
I20240502 14:05:38.360208 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.349823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:39543 (local address 127.27.174.65:38895)
0502 14:05:38.352709 (+ 2886us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.352751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.352965 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.353608 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.353640 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.353730 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:05:38.354585 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.354622 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.355837 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.355868 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.357810 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.357857 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.358194 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.358277 (+ 83us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.358397 (+ 120us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.358565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2330,"thread_start_us":1876,"threads_started":1}
I20240502 14:05:38.360950 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.350205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:38895 (local address 127.27.174.66:39543)
0502 14:05:38.351493 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.351522 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.352855 (+ 1333us) server_negotiation.cc:407] Connection header received
0502 14:05:38.353183 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.353203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.353375 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.353537 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:05:38.354814 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.355568 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.358117 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.359253 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.359390 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.359550 (+ 160us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.359732 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20240502 14:05:38.361656 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.349745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:50417 (local address 127.27.174.67:44393)
0502 14:05:38.350562 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.350599 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.352669 (+ 2070us) server_negotiation.cc:407] Connection header received
0502 14:05:38.353072 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.353107 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.353320 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.353431 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:05:38.354818 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.355596 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.358801 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.359921 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.360075 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.360259 (+ 184us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.360471 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20240502 14:05:38.362228 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.349489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44393 (local address 127.27.174.65:50417)
0502 14:05:38.352487 (+ 2998us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.352525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:38.352789 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:38.353518 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:38.353547 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:38.353650 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:05:38.354591 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.354626 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.355891 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.355930 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:38.357816 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:38.357857 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.358195 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.358274 (+ 79us) client_negotiation.cc:769] Sending connection context
0502 14:05:38.358390 (+ 116us) client_negotiation.cc:240] Negotiation successful
0502 14:05:38.358566 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2532,"spinlock_wait_cycles":25984,"thread_start_us":2599,"threads_started":1}
I20240502 14:05:38.363217 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c4259477db4245b968d771d3b5975c" candidate_uuid: "9a6739928498426a9094f8fb136eb217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" is_pre_election: true
I20240502 14:05:38.364117 371 raft_consensus.cc:2454] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a6739928498426a9094f8fb136eb217 in term 0.
I20240502 14:05:38.364006 507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c4259477db4245b968d771d3b5975c" candidate_uuid: "9a6739928498426a9094f8fb136eb217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011a57ffa47841b7af03dc586c08874a" is_pre_election: true
W20240502 14:05:38.364723 417 tablet.cc:2369] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:38.364840 507 raft_consensus.cc:2454] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a6739928498426a9094f8fb136eb217 in term 0.
I20240502 14:05:38.365589 32638 leader_election.cc:302] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7acdfe1fb0461ebf19cbe02b040f99, 9a6739928498426a9094f8fb136eb217; no voters:
I20240502 14:05:38.366401 577 raft_consensus.cc:2781] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:38.366708 577 raft_consensus.cc:482] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:38.366994 577 raft_consensus.cc:3037] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:38.373708 577 raft_consensus.cc:504] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
I20240502 14:05:38.375515 577 leader_election.cc:288] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [CANDIDATE]: Term 1 election: Requested vote from peers 011a57ffa47841b7af03dc586c08874a (127.27.174.67:44393), 7e7acdfe1fb0461ebf19cbe02b040f99 (127.27.174.66:39543)
I20240502 14:05:38.376219 507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c4259477db4245b968d771d3b5975c" candidate_uuid: "9a6739928498426a9094f8fb136eb217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011a57ffa47841b7af03dc586c08874a"
I20240502 14:05:38.376498 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c4259477db4245b968d771d3b5975c" candidate_uuid: "9a6739928498426a9094f8fb136eb217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99"
I20240502 14:05:38.376734 507 raft_consensus.cc:3037] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:38.376972 371 raft_consensus.cc:3037] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:38.383803 507 raft_consensus.cc:2454] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a6739928498426a9094f8fb136eb217 in term 1.
I20240502 14:05:38.384563 371 raft_consensus.cc:2454] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a6739928498426a9094f8fb136eb217 in term 1.
I20240502 14:05:38.384984 32640 leader_election.cc:302] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011a57ffa47841b7af03dc586c08874a, 9a6739928498426a9094f8fb136eb217; no voters:
I20240502 14:05:38.385783 577 raft_consensus.cc:2781] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:38.387624 577 raft_consensus.cc:686] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [term 1 LEADER]: Becoming Leader. State: Replica: 9a6739928498426a9094f8fb136eb217, State: Running, Role: LEADER
I20240502 14:05:38.388638 577 consensus_queue.cc:237] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } }
W20240502 14:05:38.390365 32750 tablet.cc:2369] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:38.401882 32557 catalog_manager.cc:5484] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a6739928498426a9094f8fb136eb217 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "9a6739928498426a9094f8fb136eb217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a6739928498426a9094f8fb136eb217" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:38.436844 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.430405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.27.174.65:44837)
0502 14:05:38.430887 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:38.430904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:38.430976 (+ 72us) server_negotiation.cc:407] Connection header received
0502 14:05:38.431309 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:38.431326 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:38.431551 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:38.431701 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:05:38.432343 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.433140 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:38.434454 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:38.434849 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:38.435024 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:05:38.435533 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:05:38.435670 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:38.435843 (+ 173us) server_negotiation.cc:299] Negotiation successful
0502 14:05:38.435961 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20240502 14:05:38.448683 371 raft_consensus.cc:1261] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 1 FOLLOWER]: Refusing update from remote peer 9a6739928498426a9094f8fb136eb217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:05:38.448987 507 raft_consensus.cc:1261] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 1 FOLLOWER]: Refusing update from remote peer 9a6739928498426a9094f8fb136eb217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:05:38.450237 582 consensus_queue.cc:1035] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7acdfe1fb0461ebf19cbe02b040f99" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:38.450868 577 consensus_queue.cc:1035] T f5c4259477db4245b968d771d3b5975c P 9a6739928498426a9094f8fb136eb217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011a57ffa47841b7af03dc586c08874a" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:38.470350 585 mvcc.cc:205] Tried to move back new op lower bound from 7023242192672047104 to 7023242192448884736. Current Snapshot: MvccSnapshot[applied={T|T < 7023242192672047104}]
I20240502 14:05:38.504736 32558 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51346:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:38.505246 32558 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51346:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:38.514235 32558 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 41c08ef2dd9a4ee7a96cf9ba489afc90: Sending DeleteTablet for 3 replicas of tablet f5c4259477db4245b968d771d3b5975c
I20240502 14:05:38.516239 487 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c4259477db4245b968d771d3b5975c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:40514
I20240502 14:05:38.516361 32684 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c4259477db4245b968d771d3b5975c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:58022
I20240502 14:05:38.517262 351 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c4259477db4245b968d771d3b5975c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:38 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56912
I20240502 14:05:38.526850 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32620
I20240502 14:05:38.529106 594 tablet_replica.cc:330] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: stopping tablet replica
I20240502 14:05:38.530117 594 raft_consensus.cc:2229] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:38.531052 594 raft_consensus.cc:2258] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:38.531474 596 tablet_replica.cc:330] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: stopping tablet replica
I20240502 14:05:38.532351 596 raft_consensus.cc:2229] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:38.533027 596 raft_consensus.cc:2258] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:38.533741 594 ts_tablet_manager.cc:1875] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:38.536324 596 ts_tablet_manager.cc:1875] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:38.543197 594 ts_tablet_manager.cc:1888] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:05:38.543555 594 log.cc:1196] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/f5c4259477db4245b968d771d3b5975c
I20240502 14:05:38.544142 594 ts_tablet_manager.cc:1909] T f5c4259477db4245b968d771d3b5975c P 011a57ffa47841b7af03dc586c08874a: Deleting consensus metadata
I20240502 14:05:38.547523 32546 catalog_manager.cc:4830] TS 011a57ffa47841b7af03dc586c08874a (127.27.174.67:44393): tablet f5c4259477db4245b968d771d3b5975c (table test-table [id=c5e3da1a1d554611865140bd5c681015]) successfully deleted
I20240502 14:05:38.553731 596 ts_tablet_manager.cc:1888] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:05:38.554188 596 log.cc:1196] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/f5c4259477db4245b968d771d3b5975c
I20240502 14:05:38.555004 596 ts_tablet_manager.cc:1909] T f5c4259477db4245b968d771d3b5975c P 7e7acdfe1fb0461ebf19cbe02b040f99: Deleting consensus metadata
I20240502 14:05:38.558235 32544 catalog_manager.cc:4830] TS 7e7acdfe1fb0461ebf19cbe02b040f99 (127.27.174.66:39543): tablet f5c4259477db4245b968d771d3b5975c (table test-table [id=c5e3da1a1d554611865140bd5c681015]) successfully deleted
I20240502 14:05:38.562415 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32755
W20240502 14:05:38.562705 32543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:44837 (error 108)
I20240502 14:05:38.567751 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:38.565969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44837 (local address 127.0.0.1:58028)
0502 14:05:38.566877 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:38.567133 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20240502 14:05:38.568346 32543 catalog_manager.cc:4573] TS 9a6739928498426a9094f8fb136eb217 (127.27.174.65:44837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5c4259477db4245b968d771d3b5975c: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44837: connect: Connection refused (error 111)
I20240502 14:05:38.589154 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 422
I20240502 14:05:38.612931 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 32526
2024-05-02T14:05:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (8627 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-02T14:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:38Z Disabled control of system clock
I20240502 14:05:38.702550 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36583
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240502 14:05:38.989426 601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:38.990027 601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:38.990684 601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:39.019866 601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:39.020232 601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:39.020442 601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:39.020671 601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:39.020898 601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:39.054309 601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:36583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:39.055472 601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:39.056998 601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:39.072328 607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:39.073274 610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:39.074009 601 server_base.cc:1034] running on GCE node
W20240502 14:05:39.072347 608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:40.139593 601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:40.142199 601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:40.143551 601 hybrid_clock.cc:648] HybridClock initialized: now 1714658740143526 us; error 44 us; skew 500 ppm
I20240502 14:05:40.144358 601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:40.154978 601 webserver.cc:460] Webserver started at http://127.27.174.126:39411/ using document root <none> and password file <none>
I20240502 14:05:40.155896 601 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:40.156090 601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:40.156541 601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:40.161377 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "1124123b4450474fbb62150c8d7711c3"
format_stamp: "Formatted at 2024-05-02 14:05:40 on dist-test-slave-dlb8"
server_key: "d9e708bf1145bdfe69381ecf3ff2558e"
server_key_iv: "16f44a80c541d37bcf6f6f1d967a9070"
server_key_version: "encryptionkey@0"
I20240502 14:05:40.162631 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "1124123b4450474fbb62150c8d7711c3"
format_stamp: "Formatted at 2024-05-02 14:05:40 on dist-test-slave-dlb8"
server_key: "d9e708bf1145bdfe69381ecf3ff2558e"
server_key_iv: "16f44a80c541d37bcf6f6f1d967a9070"
server_key_version: "encryptionkey@0"
I20240502 14:05:40.169518 601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:40.174820 617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:40.175781 601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:05:40.176081 601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "1124123b4450474fbb62150c8d7711c3"
format_stamp: "Formatted at 2024-05-02 14:05:40 on dist-test-slave-dlb8"
server_key: "d9e708bf1145bdfe69381ecf3ff2558e"
server_key_iv: "16f44a80c541d37bcf6f6f1d967a9070"
server_key_version: "encryptionkey@0"
I20240502 14:05:40.176424 601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:40.214532 601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:40.216003 601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:40.216465 601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:40.282835 601 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:36583
I20240502 14:05:40.282917 668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:36583 every 8 connection(s)
I20240502 14:05:40.285494 601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:40.290627 669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.294596 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 601
I20240502 14:05:40.294979 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:40.296012 28345 external_mini_cluster.cc:1428] Setting key f3cd22953b6f97d4431234e515d87fa4
I20240502 14:05:40.310959 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:40.298662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.27.174.126:36583)
0502 14:05:40.301640 (+ 2978us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:40.301661 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:40.301722 (+ 61us) server_negotiation.cc:407] Connection header received
0502 14:05:40.305648 (+ 3926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:40.305686 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:40.306030 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:40.306427 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:40.308271 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:40.308293 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:40.308319 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:40.308360 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:40.308462 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:40.308525 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:40.308766 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:40.309001 (+ 235us) server_negotiation.cc:299] Negotiation successful
0502 14:05:40.309195 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2342,"thread_start_us":1892,"threads_started":1}
I20240502 14:05:40.313184 669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: Bootstrap starting.
I20240502 14:05:40.318236 669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:40.320008 669 log.cc:824] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:40.324738 669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: No bootstrap required, opened a new log
I20240502 14:05:40.344075 669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } }
I20240502 14:05:40.344770 669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:40.344995 669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1124123b4450474fbb62150c8d7711c3, State: Initialized, Role: FOLLOWER
I20240502 14:05:40.345661 669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } }
I20240502 14:05:40.346225 669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:40.346488 669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:40.346781 669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:40.351022 669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } }
I20240502 14:05:40.351702 669 leader_election.cc:302] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1124123b4450474fbb62150c8d7711c3; no voters:
I20240502 14:05:40.353322 669 leader_election.cc:288] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:40.354089 674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:40.356832 674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1124123b4450474fbb62150c8d7711c3, State: Running, Role: LEADER
I20240502 14:05:40.357595 669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:40.357626 674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } }
I20240502 14:05:40.367286 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1124123b4450474fbb62150c8d7711c3. Latest consensus state: current_term: 1 leader_uuid: "1124123b4450474fbb62150c8d7711c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } } }
I20240502 14:05:40.368129 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:40.369443 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1124123b4450474fbb62150c8d7711c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124123b4450474fbb62150c8d7711c3" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36583 } } }
I20240502 14:05:40.370178 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:40.375727 684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:40.387040 684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:40.402729 684 catalog_manager.cc:1347] Generated new cluster ID: 0fef76457e3d4a569475efebcc30878f
I20240502 14:05:40.403084 684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:40.439383 684 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:40.440079 684 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:40.462168 684 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: Generated new TSK 0
I20240502 14:05:40.463313 684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:40.483072 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--builtin_ntp_servers=127.27.174.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240502 14:05:40.783320 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:40.783839 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:40.784411 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:40.814337 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:40.814589 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:40.815436 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:05:40.848731 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:40.849916 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:40.851428 693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:40.868590 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:40.871248 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:40.869866 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:40.872459 693 server_base.cc:1034] running on GCE node
I20240502 14:05:41.932773 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:41.935657 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:41.937188 693 hybrid_clock.cc:648] HybridClock initialized: now 1714658741937136 us; error 56 us; skew 500 ppm
I20240502 14:05:41.938342 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:41.945871 693 webserver.cc:460] Webserver started at http://127.27.174.65:41773/ using document root <none> and password file <none>
I20240502 14:05:41.946985 693 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:41.947232 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:41.947750 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:41.954787 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "9c02aafcadef4b7d991be6ac6cff1339"
format_stamp: "Formatted at 2024-05-02 14:05:41 on dist-test-slave-dlb8"
server_key: "f0ef62e39dc6bc48fa87be7ed455287f"
server_key_iv: "6b6687cc886eb8775e8c826213ec7fec"
server_key_version: "encryptionkey@0"
I20240502 14:05:41.956182 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "9c02aafcadef4b7d991be6ac6cff1339"
format_stamp: "Formatted at 2024-05-02 14:05:41 on dist-test-slave-dlb8"
server_key: "f0ef62e39dc6bc48fa87be7ed455287f"
server_key_iv: "6b6687cc886eb8775e8c826213ec7fec"
server_key_version: "encryptionkey@0"
I20240502 14:05:41.965435 693 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20240502 14:05:41.972201 709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:41.973435 693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:05:41.973845 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "9c02aafcadef4b7d991be6ac6cff1339"
format_stamp: "Formatted at 2024-05-02 14:05:41 on dist-test-slave-dlb8"
server_key: "f0ef62e39dc6bc48fa87be7ed455287f"
server_key_iv: "6b6687cc886eb8775e8c826213ec7fec"
server_key_version: "encryptionkey@0"
I20240502 14:05:41.974350 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:42.021798 693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:42.023777 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:42.024341 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:42.027374 693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:42.032590 693 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:42.032836 693 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:42.033145 693 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:42.033349 693 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:42.180289 693 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40029
I20240502 14:05:42.180335 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40029 every 8 connection(s)
I20240502 14:05:42.183573 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:05:42.187714 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 693
I20240502 14:05:42.188217 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:05:42.189251 28345 external_mini_cluster.cc:1428] Setting key dac548c9b7ec9662d0ad9454fe7f0255
I20240502 14:05:42.195163 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--builtin_ntp_servers=127.27.174.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20240502 14:05:42.201175 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:42.189867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:44817 (local address 127.27.174.126:36583)
0502 14:05:42.192181 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:42.192212 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:42.192437 (+ 225us) server_negotiation.cc:407] Connection header received
0502 14:05:42.196970 (+ 4533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:42.197006 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:42.197185 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:42.197421 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:42.198934 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:42.198967 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:42.199001 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:42.199068 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:42.199162 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:42.199222 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:42.199437 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:42.199584 (+ 147us) server_negotiation.cc:299] Negotiation successful
0502 14:05:42.199700 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"thread_start_us":1505,"threads_started":1}
I20240502 14:05:42.201138 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:42.189539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36583 (local address 127.27.174.65:44817)
0502 14:05:42.192250 (+ 2711us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:42.192312 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:42.196621 (+ 4309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:42.197606 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:42.197628 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:42.197732 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:42.198520 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:42.198534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:42.198584 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:42.198688 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:42.199367 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:42.199384 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:42.199405 (+ 21us) client_negotiation.cc:769] Sending connection context
0502 14:05:42.199591 (+ 186us) client_negotiation.cc:240] Negotiation successful
0502 14:05:42.199741 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":1513,"threads_started":1}
I20240502 14:05:42.203042 822 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36583
I20240502 14:05:42.203526 822 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:42.204464 822 heartbeater.cc:511] Master 127.27.174.126:36583 requested a full tablet report, sending...
I20240502 14:05:42.207059 634 ts_manager.cc:194] Registered new tserver with Master: 9c02aafcadef4b7d991be6ac6cff1339 (127.27.174.65:40029)
W20240502 14:05:42.481499 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:42.482023 826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:42.482641 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:42.511325 826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:42.511548 826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:42.512388 826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:05:42.544425 826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:42.545598 826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:42.547128 826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:42.563151 835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:43.210294 822 heartbeater.cc:503] Master 127.27.174.126:36583 was elected leader, sending a full tablet report...
W20240502 14:05:42.565022 833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:42.565496 826 server_base.cc:1034] running on GCE node
W20240502 14:05:42.563891 832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:43.613405 826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:43.615819 826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:43.617197 826 hybrid_clock.cc:648] HybridClock initialized: now 1714658743617155 us; error 61 us; skew 500 ppm
I20240502 14:05:43.617956 826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:43.624092 826 webserver.cc:460] Webserver started at http://127.27.174.66:40533/ using document root <none> and password file <none>
I20240502 14:05:43.625000 826 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:43.625216 826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:43.625639 826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:43.630702 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "92cd204414354bbd874deb633cc31507"
format_stamp: "Formatted at 2024-05-02 14:05:43 on dist-test-slave-dlb8"
server_key: "b8a9481ff5c0d1f0a09452f064b10586"
server_key_iv: "db1f7f80e25152d22f962364ea23e616"
server_key_version: "encryptionkey@0"
I20240502 14:05:43.631798 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "92cd204414354bbd874deb633cc31507"
format_stamp: "Formatted at 2024-05-02 14:05:43 on dist-test-slave-dlb8"
server_key: "b8a9481ff5c0d1f0a09452f064b10586"
server_key_iv: "db1f7f80e25152d22f962364ea23e616"
server_key_version: "encryptionkey@0"
I20240502 14:05:43.638823 826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 14:05:43.644318 842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:43.645274 826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:05:43.645577 826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "92cd204414354bbd874deb633cc31507"
format_stamp: "Formatted at 2024-05-02 14:05:43 on dist-test-slave-dlb8"
server_key: "b8a9481ff5c0d1f0a09452f064b10586"
server_key_iv: "db1f7f80e25152d22f962364ea23e616"
server_key_version: "encryptionkey@0"
I20240502 14:05:43.645936 826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:43.684000 826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:43.685755 826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:43.686239 826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:43.688622 826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:43.692603 826 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:43.692798 826 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:43.693014 826 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:43.693159 826 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:43.820905 826 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45553
I20240502 14:05:43.820996 954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45553 every 8 connection(s)
I20240502 14:05:43.823400 826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:05:43.830902 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 826
I20240502 14:05:43.831310 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:05:43.832229 28345 external_mini_cluster.cc:1428] Setting key 92836235dfeafbda8abe78da4e9b2fac
I20240502 14:05:43.837724 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--builtin_ntp_servers=127.27.174.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20240502 14:05:43.839627 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:43.827632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36583 (local address 127.27.174.66:37929)
0502 14:05:43.830524 (+ 2892us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:43.830591 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:43.835124 (+ 4533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:43.836063 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:43.836086 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:43.836203 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:43.836998 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:43.837012 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:43.837062 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:43.837166 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:43.837808 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:43.837826 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:43.837843 (+ 17us) client_negotiation.cc:769] Sending connection context
0502 14:05:43.838077 (+ 234us) client_negotiation.cc:240] Negotiation successful
0502 14:05:43.838319 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":1619,"threads_started":1}
I20240502 14:05:43.839897 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:43.827384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37929 (local address 127.27.174.126:36583)
0502 14:05:43.828462 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:43.828481 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:43.830756 (+ 2275us) server_negotiation.cc:407] Connection header received
0502 14:05:43.835464 (+ 4708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:43.835500 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:43.835666 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:43.835837 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:43.837415 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:43.837442 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:43.837470 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:43.837523 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:43.837614 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:43.837664 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:43.837874 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:43.838026 (+ 152us) server_negotiation.cc:299] Negotiation successful
0502 14:05:43.838212 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":432,"threads_started":1}
I20240502 14:05:43.841624 955 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36583
I20240502 14:05:43.842006 955 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:43.842742 955 heartbeater.cc:511] Master 127.27.174.126:36583 requested a full tablet report, sending...
I20240502 14:05:43.844749 634 ts_manager.cc:194] Registered new tserver with Master: 92cd204414354bbd874deb633cc31507 (127.27.174.66:45553)
W20240502 14:05:44.123610 959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:44.124091 959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:44.124630 959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:44.153843 959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:44.154139 959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:44.154958 959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:05:44.194553 959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:44.195626 959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:44.197070 959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:44.212635 965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:44.847484 955 heartbeater.cc:503] Master 127.27.174.126:36583 was elected leader, sending a full tablet report...
W20240502 14:05:44.213428 968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:44.214682 959 server_base.cc:1034] running on GCE node
W20240502 14:05:44.214604 966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:45.255736 959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:45.257808 959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:45.259125 959 hybrid_clock.cc:648] HybridClock initialized: now 1714658745259097 us; error 75 us; skew 500 ppm
I20240502 14:05:45.259882 959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:45.269896 959 webserver.cc:460] Webserver started at http://127.27.174.67:46423/ using document root <none> and password file <none>
I20240502 14:05:45.270766 959 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:45.270939 959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:45.271322 959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:45.276258 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "bb9f23be1c6541ca8d08b3682ecc3349"
format_stamp: "Formatted at 2024-05-02 14:05:45 on dist-test-slave-dlb8"
server_key: "35f811fc78a61a8cbb34e72a2aeac31e"
server_key_iv: "32b7b294deaa6cf9c3cb8b5a3c41a42b"
server_key_version: "encryptionkey@0"
I20240502 14:05:45.277307 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9f23be1c6541ca8d08b3682ecc3349"
format_stamp: "Formatted at 2024-05-02 14:05:45 on dist-test-slave-dlb8"
server_key: "35f811fc78a61a8cbb34e72a2aeac31e"
server_key_iv: "32b7b294deaa6cf9c3cb8b5a3c41a42b"
server_key_version: "encryptionkey@0"
I20240502 14:05:45.284679 959 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240502 14:05:45.290153 975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:45.291249 959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240502 14:05:45.291534 959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "bb9f23be1c6541ca8d08b3682ecc3349"
format_stamp: "Formatted at 2024-05-02 14:05:45 on dist-test-slave-dlb8"
server_key: "35f811fc78a61a8cbb34e72a2aeac31e"
server_key_iv: "32b7b294deaa6cf9c3cb8b5a3c41a42b"
server_key_version: "encryptionkey@0"
I20240502 14:05:45.291918 959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:45.329092 959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:45.330547 959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:45.330955 959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:45.333278 959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:45.337378 959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:45.337570 959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:45.337800 959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:45.337944 959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:45.464241 959 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:37455
I20240502 14:05:45.464344 1087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:37455 every 8 connection(s)
I20240502 14:05:45.466753 959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:05:45.476107 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 959
I20240502 14:05:45.476497 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:05:45.477480 28345 external_mini_cluster.cc:1428] Setting key 1fd23bd6528c30a6911ecd0000c0e934
I20240502 14:05:45.482439 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.470499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37097 (local address 127.27.174.126:36583)
0502 14:05:45.471570 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.471587 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.473844 (+ 2257us) server_negotiation.cc:407] Connection header received
0502 14:05:45.478338 (+ 4494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.478368 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.478568 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.478711 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.480413 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.480433 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.480448 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.480485 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.480559 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.480610 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.480806 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.481008 (+ 202us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.481104 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":414,"threads_started":1}
I20240502 14:05:45.483311 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.470802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36583 (local address 127.27.174.67:37097)
0502 14:05:45.473644 (+ 2842us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.473710 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.477928 (+ 4218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.478972 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.478995 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.479106 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.479998 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.480013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.480066 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.480176 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.480775 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.480796 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.480813 (+ 17us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.481071 (+ 258us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.481277 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"thread_start_us":1598,"threads_started":1}
I20240502 14:05:45.484956 1088 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36583
I20240502 14:05:45.485268 1088 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:45.485910 1088 heartbeater.cc:511] Master 127.27.174.126:36583 requested a full tablet report, sending...
I20240502 14:05:45.487731 634 ts_manager.cc:194] Registered new tserver with Master: bb9f23be1c6541ca8d08b3682ecc3349 (127.27.174.67:37455)
I20240502 14:05:45.497061 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:05:45.513140 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.508256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.27.174.126:36583)
0502 14:05:45.508763 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.508785 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.509677 (+ 892us) server_negotiation.cc:407] Connection header received
0502 14:05:45.509995 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.510013 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.510186 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.510333 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.511587 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.511605 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.511624 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.511660 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.511720 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.511748 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.511923 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.512133 (+ 210us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.512261 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20240502 14:05:45.518923 634 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:05:45.538231 634 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:05:45.580464 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.564368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45553 (local address 127.0.0.1:45948)
0502 14:05:45.571730 (+ 7362us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.571951 (+ 221us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.572145 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.576285 (+ 4140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.576326 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.576462 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.577285 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.577318 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.577401 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.577516 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.578550 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.578598 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.578640 (+ 42us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.579142 (+ 502us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.579250 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6761,"thread_start_us":6111,"threads_started":1}
I20240502 14:05:45.582823 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.562402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.27.174.65:40029)
0502 14:05:45.573160 (+ 10758us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.573189 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.573257 (+ 68us) server_negotiation.cc:407] Connection header received
0502 14:05:45.573436 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.573464 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.573735 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.573945 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.578036 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.578089 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.578128 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.578187 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.578426 (+ 239us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.578491 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.580920 (+ 2429us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.581049 (+ 129us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.581191 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9984,"thread_start_us":9786,"threads_started":1}
I20240502 14:05:45.583179 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.562266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40029 (local address 127.0.0.1:59370)
0502 14:05:45.567198 (+ 4932us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.567295 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.567608 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.574154 (+ 6546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.574205 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.574393 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.575679 (+ 1286us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.575715 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.575827 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.576011 (+ 184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.578693 (+ 2682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.578727 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.578757 (+ 30us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.578975 (+ 218us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.579143 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4022,"thread_start_us":16472,"threads_started":1}
I20240502 14:05:45.582906 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.568503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.27.174.66:45553)
0502 14:05:45.573270 (+ 4767us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.573296 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.573341 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:05:45.573500 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.573524 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.573737 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.573945 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.577778 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.577811 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.577845 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.577894 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.577999 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.578041 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.578402 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.581123 (+ 2721us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.581278 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4189,"thread_start_us":3838,"threads_started":1}
I20240502 14:05:45.586726 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.566642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37455 (local address 127.0.0.1:55562)
0502 14:05:45.580084 (+ 13442us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.580141 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.580332 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.582586 (+ 2254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.582611 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.582697 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.583485 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.583509 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.583580 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.583693 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.584743 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.584775 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.584800 (+ 25us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.585063 (+ 263us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.585168 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12860,"thread_start_us":4353,"threads_started":1}
I20240502 14:05:45.586869 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.570329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.27.174.67:37455)
0502 14:05:45.578492 (+ 8163us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.578523 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.580282 (+ 1759us) server_negotiation.cc:407] Connection header received
0502 14:05:45.580576 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.580606 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.580857 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.581034 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.583994 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.584027 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.584073 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.584137 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.584267 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.584321 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.584587 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.585073 (+ 486us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.585213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7508,"thread_start_us":6983,"threads_started":1}
I20240502 14:05:45.587663 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 (DEFAULT_TABLE table=test-table [id=809526effe3d4386b937fc1016bdb214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:45.587698 757 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 (DEFAULT_TABLE table=test-table [id=809526effe3d4386b937fc1016bdb214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:45.589223 890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb05a8b9ae44cff8649a99624dd6669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:45.590243 757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb05a8b9ae44cff8649a99624dd6669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:45.591773 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 (DEFAULT_TABLE table=test-table [id=809526effe3d4386b937fc1016bdb214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:45.593699 1023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb05a8b9ae44cff8649a99624dd6669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:45.611115 1107 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Bootstrap starting.
I20240502 14:05:45.618750 1107 tablet_bootstrap.cc:654] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:45.621109 1108 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Bootstrap starting.
I20240502 14:05:45.621309 1107 log.cc:824] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:45.623736 1109 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Bootstrap starting.
I20240502 14:05:45.627943 1108 tablet_bootstrap.cc:654] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:45.629467 1107 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: No bootstrap required, opened a new log
I20240502 14:05:45.629994 1107 ts_tablet_manager.cc:1367] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20240502 14:05:45.630185 1109 tablet_bootstrap.cc:654] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:45.630263 1108 log.cc:824] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:45.631944 1109 log.cc:824] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:45.636592 1108 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: No bootstrap required, opened a new log
I20240502 14:05:45.637110 1108 ts_tablet_manager.cc:1367] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240502 14:05:45.640276 1109 tablet_bootstrap.cc:492] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: No bootstrap required, opened a new log
I20240502 14:05:45.640691 1109 ts_tablet_manager.cc:1367] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20240502 14:05:45.661916 1109 raft_consensus.cc:348] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.662011 1107 raft_consensus.cc:348] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.662815 1109 raft_consensus.cc:374] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:45.662931 1107 raft_consensus.cc:374] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:45.663100 1109 raft_consensus.cc:729] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f23be1c6541ca8d08b3682ecc3349, State: Initialized, Role: FOLLOWER
I20240502 14:05:45.663265 1107 raft_consensus.cc:729] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cd204414354bbd874deb633cc31507, State: Initialized, Role: FOLLOWER
I20240502 14:05:45.664026 1109 consensus_queue.cc:260] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.664139 1107 consensus_queue.cc:260] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.667204 1088 heartbeater.cc:503] Master 127.27.174.126:36583 was elected leader, sending a full tablet report...
I20240502 14:05:45.668490 1109 ts_tablet_manager.cc:1398] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Time spent starting tablet: real 0.028s user 0.022s sys 0.006s
I20240502 14:05:45.668853 1108 raft_consensus.cc:348] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.669843 1108 raft_consensus.cc:374] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:45.670150 1108 raft_consensus.cc:729] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c02aafcadef4b7d991be6ac6cff1339, State: Initialized, Role: FOLLOWER
I20240502 14:05:45.671016 1107 ts_tablet_manager.cc:1398] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Time spent starting tablet: real 0.041s user 0.028s sys 0.012s
I20240502 14:05:45.671197 1108 consensus_queue.cc:260] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.675947 1108 ts_tablet_manager.cc:1398] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Time spent starting tablet: real 0.039s user 0.021s sys 0.016s
W20240502 14:05:45.692373 823 tablet.cc:2369] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:05:45.721069 1089 tablet.cc:2369] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:05:45.829459 956 tablet.cc:2369] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:45.936300 1115 raft_consensus.cc:482] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:45.936775 1115 raft_consensus.cc:504] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.939241 1115 leader_election.cc:288] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cd204414354bbd874deb633cc31507 (127.27.174.66:45553), bb9f23be1c6541ca8d08b3682ecc3349 (127.27.174.67:37455)
I20240502 14:05:45.947227 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.940293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:55275 (local address 127.27.174.67:37455)
0502 14:05:45.941150 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.941182 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.943017 (+ 1835us) server_negotiation.cc:407] Connection header received
0502 14:05:45.943331 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.943354 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.943557 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.943743 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.945116 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.945148 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.945171 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.945237 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.945310 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.945345 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.945486 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.945727 (+ 241us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.945854 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20240502 14:05:45.947388 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.939805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42223 (local address 127.27.174.66:45553)
0502 14:05:45.940601 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:45.940633 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:45.943019 (+ 2386us) server_negotiation.cc:407] Connection header received
0502 14:05:45.943507 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:45.943526 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:45.943650 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:45.943808 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:45.945320 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:45.945341 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:45.945355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:45.945390 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:45.945470 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:45.945519 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:45.945722 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:45.945928 (+ 206us) server_negotiation.cc:299] Negotiation successful
0502 14:05:45.946064 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:05:45.947801 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.939964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:37455 (local address 127.27.174.65:55275)
0502 14:05:45.942818 (+ 2854us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.942867 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.943126 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.943828 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.943857 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.943953 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.944766 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.944780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.944828 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.944901 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.945497 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.945523 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.945548 (+ 25us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.945752 (+ 204us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.945853 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2233,"thread_start_us":1879,"threads_started":1}
I20240502 14:05:45.948071 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:45.939718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45553 (local address 127.27.174.65:42223)
0502 14:05:45.942807 (+ 3089us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:45.942866 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:45.943202 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:45.943952 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:45.943972 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:45.944044 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:05:45.944934 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:05:45.944950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:05:45.945003 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:05:45.945101 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:05:45.945685 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:05:45.945712 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:05:45.945735 (+ 23us) client_negotiation.cc:769] Sending connection context
0502 14:05:45.945946 (+ 211us) client_negotiation.cc:240] Negotiation successful
0502 14:05:45.946070 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"spinlock_wait_cycles":118144,"thread_start_us":1874,"threads_started":1}
I20240502 14:05:45.949389 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb05a8b9ae44cff8649a99624dd6669" candidate_uuid: "9c02aafcadef4b7d991be6ac6cff1339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" is_pre_election: true
I20240502 14:05:45.949441 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb05a8b9ae44cff8649a99624dd6669" candidate_uuid: "9c02aafcadef4b7d991be6ac6cff1339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cd204414354bbd874deb633cc31507" is_pre_election: true
I20240502 14:05:45.950083 1043 raft_consensus.cc:2454] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c02aafcadef4b7d991be6ac6cff1339 in term 0.
I20240502 14:05:45.950085 910 raft_consensus.cc:2454] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c02aafcadef4b7d991be6ac6cff1339 in term 0.
I20240502 14:05:45.951114 711 leader_election.cc:302] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cd204414354bbd874deb633cc31507, 9c02aafcadef4b7d991be6ac6cff1339; no voters:
I20240502 14:05:45.951932 1115 raft_consensus.cc:2781] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:45.952255 1115 raft_consensus.cc:482] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:45.952500 1115 raft_consensus.cc:3037] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:45.956800 1115 raft_consensus.cc:504] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.958263 1115 leader_election.cc:288] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [CANDIDATE]: Term 1 election: Requested vote from peers 92cd204414354bbd874deb633cc31507 (127.27.174.66:45553), bb9f23be1c6541ca8d08b3682ecc3349 (127.27.174.67:37455)
I20240502 14:05:45.958914 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb05a8b9ae44cff8649a99624dd6669" candidate_uuid: "9c02aafcadef4b7d991be6ac6cff1339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cd204414354bbd874deb633cc31507"
I20240502 14:05:45.959180 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb05a8b9ae44cff8649a99624dd6669" candidate_uuid: "9c02aafcadef4b7d991be6ac6cff1339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f23be1c6541ca8d08b3682ecc3349"
I20240502 14:05:45.959376 910 raft_consensus.cc:3037] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:45.959626 1043 raft_consensus.cc:3037] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:45.963848 910 raft_consensus.cc:2454] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c02aafcadef4b7d991be6ac6cff1339 in term 1.
I20240502 14:05:45.963894 1043 raft_consensus.cc:2454] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c02aafcadef4b7d991be6ac6cff1339 in term 1.
I20240502 14:05:45.964774 713 leader_election.cc:302] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c02aafcadef4b7d991be6ac6cff1339, bb9f23be1c6541ca8d08b3682ecc3349; no voters:
I20240502 14:05:45.965375 1115 raft_consensus.cc:2781] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:45.966827 1115 raft_consensus.cc:686] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [term 1 LEADER]: Becoming Leader. State: Replica: 9c02aafcadef4b7d991be6ac6cff1339, State: Running, Role: LEADER
I20240502 14:05:45.967525 1115 consensus_queue.cc:237] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } }
I20240502 14:05:45.978195 634 catalog_manager.cc:5484] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c02aafcadef4b7d991be6ac6cff1339 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "9c02aafcadef4b7d991be6ac6cff1339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c02aafcadef4b7d991be6ac6cff1339" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:46.056529 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:46.051936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.27.174.65:40029)
0502 14:05:46.052703 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:46.052730 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:46.053145 (+ 415us) server_negotiation.cc:407] Connection header received
0502 14:05:46.053533 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:46.053555 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:46.053736 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:46.053865 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:46.055068 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:46.055086 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:46.055103 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:46.055141 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:46.055197 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:46.055226 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:46.055367 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:46.055514 (+ 147us) server_negotiation.cc:299] Negotiation successful
0502 14:05:46.055597 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:05:46.067716 910 raft_consensus.cc:1261] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 1 FOLLOWER]: Refusing update from remote peer 9c02aafcadef4b7d991be6ac6cff1339: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:05:46.068032 1043 raft_consensus.cc:1261] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 1 FOLLOWER]: Refusing update from remote peer 9c02aafcadef4b7d991be6ac6cff1339: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:05:46.069134 1118 consensus_queue.cc:1035] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cd204414354bbd874deb633cc31507" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:46.069674 1115 consensus_queue.cc:1035] T 0bb05a8b9ae44cff8649a99624dd6669 P 9c02aafcadef4b7d991be6ac6cff1339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9f23be1c6541ca8d08b3682ecc3349" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:46.089706 1123 mvcc.cc:205] Tried to move back new op lower bound from 7023242223880384512 to 7023242223489593344. Current Snapshot: MvccSnapshot[applied={T|T < 7023242223880384512}]
I20240502 14:05:46.090945 1122 mvcc.cc:205] Tried to move back new op lower bound from 7023242223880384512 to 7023242223489593344. Current Snapshot: MvccSnapshot[applied={T|T < 7023242223880384512}]
I20240502 14:05:46.118619 632 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36154:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:46.119258 632 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36154:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:05:46.127508 632 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1124123b4450474fbb62150c8d7711c3: Sending DeleteTablet for 3 replicas of tablet 0bb05a8b9ae44cff8649a99624dd6669
I20240502 14:05:46.129215 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:46 UTC) from {username='slave'} at 127.0.0.1:59370
I20240502 14:05:46.129566 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:46 UTC) from {username='slave'} at 127.0.0.1:45948
I20240502 14:05:46.130959 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb05a8b9ae44cff8649a99624dd6669 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:05:46 UTC) from {username='slave'} at 127.0.0.1:55562
I20240502 14:05:46.132926 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 693
I20240502 14:05:46.134728 1133 tablet_replica.cc:330] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: stopping tablet replica
I20240502 14:05:46.135602 1133 raft_consensus.cc:2229] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:46.136081 1134 tablet_replica.cc:330] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: stopping tablet replica
I20240502 14:05:46.136432 1133 raft_consensus.cc:2258] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:46.136826 1134 raft_consensus.cc:2229] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:46.137418 1134 raft_consensus.cc:2258] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:46.139847 1133 ts_tablet_manager.cc:1875] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:46.140247 1134 ts_tablet_manager.cc:1875] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:05:46.154892 1133 ts_tablet_manager.cc:1888] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:05:46.154891 1134 ts_tablet_manager.cc:1888] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:05:46.155315 1133 log.cc:1196] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/0bb05a8b9ae44cff8649a99624dd6669
I20240502 14:05:46.155316 1134 log.cc:1196] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/0bb05a8b9ae44cff8649a99624dd6669
I20240502 14:05:46.156149 1134 ts_tablet_manager.cc:1909] T 0bb05a8b9ae44cff8649a99624dd6669 P bb9f23be1c6541ca8d08b3682ecc3349: Deleting consensus metadata
I20240502 14:05:46.156222 1133 ts_tablet_manager.cc:1909] T 0bb05a8b9ae44cff8649a99624dd6669 P 92cd204414354bbd874deb633cc31507: Deleting consensus metadata
I20240502 14:05:46.159086 621 catalog_manager.cc:4830] TS bb9f23be1c6541ca8d08b3682ecc3349 (127.27.174.67:37455): tablet 0bb05a8b9ae44cff8649a99624dd6669 (table test-table [id=809526effe3d4386b937fc1016bdb214]) successfully deleted
I20240502 14:05:46.159232 619 catalog_manager.cc:4830] TS 92cd204414354bbd874deb633cc31507 (127.27.174.66:45553): tablet 0bb05a8b9ae44cff8649a99624dd6669 (table test-table [id=809526effe3d4386b937fc1016bdb214]) successfully deleted
W20240502 14:05:46.161907 618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:40029 (error 108)
I20240502 14:05:46.162022 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 826
I20240502 14:05:46.169696 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:46.167668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40029 (local address 127.0.0.1:59388)
0502 14:05:46.168820 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:46.168995 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":524,"threads_started":1}
W20240502 14:05:46.170150 618 catalog_manager.cc:4573] TS 9c02aafcadef4b7d991be6ac6cff1339 (127.27.174.65:40029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bb05a8b9ae44cff8649a99624dd6669: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40029: connect: Connection refused (error 111)
I20240502 14:05:46.188678 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 959
I20240502 14:05:46.212934 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 601
2024-05-02T14:05:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (7593 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:540: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestJwtMiniCluster (6 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240502 14:05:46.272612 28345 mini_oidc.cc:152] Starting JWKS server
I20240502 14:05:46.276254 28345 webserver.cc:460] Webserver started at https://127.0.0.1:40095/ using document root <none> and password file <none>
I20240502 14:05:46.276762 28345 mini_oidc.cc:189] Starting OIDC Discovery server
I20240502 14:05:46.277784 28345 webserver.cc:460] Webserver started at http://127.0.0.1:34117/ using document root <none> and password file <none>
2024-05-02T14:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:46Z Disabled control of system clock
I20240502 14:05:46.305812 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37721
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37721
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/testchainca.cert with env {}
W20240502 14:05:46.590410 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:46.591022 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:46.591580 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:46.620375 1142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:46.620760 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:46.620987 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:46.621304 1142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:46.621631 1142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240502 14:05:46.621985 1142 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240502 14:05:46.622223 1142 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40095/jwks.json
I20240502 14:05:46.657213 1142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:46.658300 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:46.659256 1142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:46.673049 1148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:46.674427 1149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:46.675370 1142 server_base.cc:1034] running on GCE node
W20240502 14:05:46.676285 1151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.722796 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:47.725659 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:47.727053 1142 hybrid_clock.cc:648] HybridClock initialized: now 1714658747727025 us; error 46 us; skew 500 ppm
I20240502 14:05:47.727851 1142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:47.734834 1142 webserver.cc:460] Webserver started at http://127.27.174.126:38663/ using document root <none> and password file <none>
I20240502 14:05:47.735738 1142 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:47.735939 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:47.736380 1142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:47.741509 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-dlb8"
server_key: "8f40bfb334acb154652516a0e0e7ede1"
server_key_iv: "823339dc14c88c82ed6418a07ca15bf8"
server_key_version: "encryptionkey@0"
I20240502 14:05:47.742745 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-dlb8"
server_key: "8f40bfb334acb154652516a0e0e7ede1"
server_key_iv: "823339dc14c88c82ed6418a07ca15bf8"
server_key_version: "encryptionkey@0"
I20240502 14:05:47.749852 1142 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:47.755417 1159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.756434 1142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:05:47.756762 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-dlb8"
server_key: "8f40bfb334acb154652516a0e0e7ede1"
server_key_iv: "823339dc14c88c82ed6418a07ca15bf8"
server_key_version: "encryptionkey@0"
I20240502 14:05:47.757128 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:47.806602 1142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:47.807991 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:47.808415 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:47.875208 1142 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:37721
I20240502 14:05:47.875283 1210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:37721 every 8 connection(s)
I20240502 14:05:47.878187 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:47.880806 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1142
I20240502 14:05:47.881661 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:47.882910 28345 external_mini_cluster.cc:1428] Setting key a56a95991e869b7e4f0f3c8acacdc7cb
I20240502 14:05:47.883778 1211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:47.905596 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:47.885141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.27.174.126:37721)
0502 14:05:47.887972 (+ 2831us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:47.888010 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:47.888085 (+ 75us) server_negotiation.cc:407] Connection header received
0502 14:05:47.894028 (+ 5943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:47.894103 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:47.894652 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:47.895136 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:47.896435 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:47.898540 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:47.899876 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:47.900476 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:47.901058 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:47.901176 (+ 118us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:47.901231 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:47.902287 (+ 1056us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:47.902412 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:47.902467 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:47.902642 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:47.902892 (+ 250us) server_negotiation.cc:299] Negotiation successful
0502 14:05:47.903194 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1981,"thread_start_us":1644,"threads_started":1}
I20240502 14:05:47.909857 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:05:47.910193 1211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84: Bootstrap starting.
I20240502 14:05:47.914817 1211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:47.916581 1211 log.cc:824] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:47.921377 1211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84: No bootstrap required, opened a new log
I20240502 14:05:47.941195 1211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } }
I20240502 14:05:47.941910 1211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:47.942162 1211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18156e4a7fff4cf39e3bdcd8d34b7e84, State: Initialized, Role: FOLLOWER
I20240502 14:05:47.942770 1211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } }
I20240502 14:05:47.943253 1211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:47.943482 1211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:47.943756 1211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:47.947687 1211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } }
I20240502 14:05:47.948498 1211 leader_election.cc:302] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18156e4a7fff4cf39e3bdcd8d34b7e84; no voters:
I20240502 14:05:47.950351 1211 leader_election.cc:288] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:47.950690 1216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:47.953162 1216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [term 1 LEADER]: Becoming Leader. State: Replica: 18156e4a7fff4cf39e3bdcd8d34b7e84, State: Running, Role: LEADER
I20240502 14:05:47.953907 1216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } }
I20240502 14:05:47.954859 1211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:47.964296 1217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } } }
I20240502 14:05:47.965080 1217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:47.966279 1218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18156e4a7fff4cf39e3bdcd8d34b7e84. Latest consensus state: current_term: 1 leader_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18156e4a7fff4cf39e3bdcd8d34b7e84" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37721 } } }
I20240502 14:05:47.966989 1218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:47.968650 1225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:47.980454 1225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:47.995191 1225 catalog_manager.cc:1347] Generated new cluster ID: 907288dd07ee48ee93e0dacd277cd2f1
I20240502 14:05:47.995512 1225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:48.007683 1225 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:48.009688 1225 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:48.025270 1225 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 18156e4a7fff4cf39e3bdcd8d34b7e84: Generated new TSK 0
I20240502 14:05:48.026214 1225 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240502 14:05:48.083057 1241 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240502 14:05:48.083401 1241 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240502 14:05:48.085410 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:48.051881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.27.174.126:37721)
0502 14:05:48.052365 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:48.052380 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:48.054174 (+ 1794us) server_negotiation.cc:407] Connection header received
0502 14:05:48.054638 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:48.054658 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:48.054870 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:48.055085 (+ 215us) server_negotiation.cc:226] Negotiated authn=JWT
0502 14:05:48.056257 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:48.057061 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:48.058527 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:48.059048 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:48.059245 (+ 197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0502 14:05:48.083421 (+ 24176us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0502 14:05:48.084002 (+ 581us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":78}
W20240502 14:05:48.085692 1214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240502 14:05:48.086740 1240 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:05:48.051823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:37721 (local address 127.0.0.1:36066)
0502 14:05:48.053973 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:48.054013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:48.054349 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:48.055133 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:48.055167 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:48.055225 (+ 58us) client_negotiation.cc:189] Negotiated authn=JWT
0502 14:05:48.055991 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:48.056023 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:48.057291 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:48.057315 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:48.058288 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:48.058325 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:48.058629 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:48.058766 (+ 137us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0502 14:05:48.084019 (+ 25253us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0502 14:05:48.084457 (+ 438us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:37721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1614,"thread_start_us":1282,"threads_started":1}
I20240502 14:05:48.088304 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1142
2024-05-02T14:05:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (2877 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240502 14:05:49.151306 28345 mini_oidc.cc:152] Starting JWKS server
I20240502 14:05:49.154620 28345 webserver.cc:460] Webserver started at https://127.0.0.1:35057/ using document root <none> and password file <none>
I20240502 14:05:49.155146 28345 mini_oidc.cc:189] Starting OIDC Discovery server
I20240502 14:05:49.156790 28345 webserver.cc:460] Webserver started at http://127.0.0.1:39811/ using document root <none> and password file <none>
2024-05-02T14:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:49Z Disabled control of system clock
I20240502 14:05:49.186024 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46269
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35057/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240502 14:05:49.474103 1248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:49.474663 1248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:49.475175 1248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:49.502871 1248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:49.503168 1248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:49.503316 1248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:49.503505 1248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:49.503752 1248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240502 14:05:49.503993 1248 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240502 14:05:49.504184 1248 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35057/jwks.json
I20240502 14:05:49.536778 1248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35057/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:49.537922 1248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:49.539449 1248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:49.555449 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:49.556710 1248 server_base.cc:1034] running on GCE node
W20240502 14:05:49.555507 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:49.555697 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:50.608139 1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:50.611351 1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:50.612789 1248 hybrid_clock.cc:648] HybridClock initialized: now 1714658750612752 us; error 59 us; skew 500 ppm
I20240502 14:05:50.613606 1248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:50.625428 1248 webserver.cc:460] Webserver started at http://127.27.174.126:40237/ using document root <none> and password file <none>
I20240502 14:05:50.626471 1248 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:50.626703 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:50.627166 1248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:50.632470 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "850eb592446b46698e196538faa6571d"
format_stamp: "Formatted at 2024-05-02 14:05:50 on dist-test-slave-dlb8"
server_key: "6c256e64a3a15594658bcdc2bb974998"
server_key_iv: "0979b9435a2590eaa03e5ce3d201b2ec"
server_key_version: "encryptionkey@0"
I20240502 14:05:50.633641 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "850eb592446b46698e196538faa6571d"
format_stamp: "Formatted at 2024-05-02 14:05:50 on dist-test-slave-dlb8"
server_key: "6c256e64a3a15594658bcdc2bb974998"
server_key_iv: "0979b9435a2590eaa03e5ce3d201b2ec"
server_key_version: "encryptionkey@0"
I20240502 14:05:50.641885 1248 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:05:50.647635 1264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:50.648725 1248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:05:50.649073 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "850eb592446b46698e196538faa6571d"
format_stamp: "Formatted at 2024-05-02 14:05:50 on dist-test-slave-dlb8"
server_key: "6c256e64a3a15594658bcdc2bb974998"
server_key_iv: "0979b9435a2590eaa03e5ce3d201b2ec"
server_key_version: "encryptionkey@0"
I20240502 14:05:50.649466 1248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:50.732991 1248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:50.734521 1248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:50.734977 1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:50.802870 1248 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46269
I20240502 14:05:50.802942 1315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46269 every 8 connection(s)
I20240502 14:05:50.805578 1248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:50.810523 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:50.811069 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1248
I20240502 14:05:50.811492 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:50.812538 28345 external_mini_cluster.cc:1428] Setting key 460f444e898b7fbe4fa1e7e891bd63b2
I20240502 14:05:50.833278 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:50.814521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.27.174.126:46269)
0502 14:05:50.817311 (+ 2790us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:50.817348 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:50.817424 (+ 76us) server_negotiation.cc:407] Connection header received
0502 14:05:50.821501 (+ 4077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:50.821554 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:50.822134 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:50.822609 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:50.823821 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.826107 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:50.827562 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.828160 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:50.828781 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:50.828898 (+ 117us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:50.828942 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:50.829969 (+ 1027us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:50.830123 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:50.830180 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:50.830351 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:50.830603 (+ 252us) server_negotiation.cc:299] Negotiation successful
0502 14:05:50.830917 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1955,"thread_start_us":1591,"threads_started":1}
I20240502 14:05:50.837834 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:05:50.838326 1316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d: Bootstrap starting.
I20240502 14:05:50.843185 1316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:50.845001 1316 log.cc:824] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:50.849861 1316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d: No bootstrap required, opened a new log
I20240502 14:05:50.870117 1316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } }
I20240502 14:05:50.870949 1316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:50.871165 1316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850eb592446b46698e196538faa6571d, State: Initialized, Role: FOLLOWER
I20240502 14:05:50.871798 1316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } }
I20240502 14:05:50.872256 1316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:50.872505 1316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:50.872766 1316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:50.876598 1316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } }
I20240502 14:05:50.877231 1316 leader_election.cc:302] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850eb592446b46698e196538faa6571d; no voters:
I20240502 14:05:50.878914 1316 leader_election.cc:288] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:50.879670 1321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:50.881707 1321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [term 1 LEADER]: Becoming Leader. State: Replica: 850eb592446b46698e196538faa6571d, State: Running, Role: LEADER
I20240502 14:05:50.882453 1321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } }
I20240502 14:05:50.882882 1316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:50.893062 1322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850eb592446b46698e196538faa6571d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } } }
I20240502 14:05:50.893321 1323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850eb592446b46698e196538faa6571d. Latest consensus state: current_term: 1 leader_uuid: "850eb592446b46698e196538faa6571d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850eb592446b46698e196538faa6571d" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46269 } } }
I20240502 14:05:50.893961 1322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:50.894258 1323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:50.897312 1331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:50.909783 1331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:50.925379 1331 catalog_manager.cc:1347] Generated new cluster ID: cc09c252811548c0b2b218d93c280829
I20240502 14:05:50.925640 1331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:50.939918 1331 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:50.941277 1331 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:50.957708 1331 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 850eb592446b46698e196538faa6571d: Generated new TSK 0
I20240502 14:05:50.958808 1331 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240502 14:05:51.027460 1346 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240502 14:05:51.027748 1346 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240502 14:05:51.031847 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:50.984749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.27.174.126:46269)
0502 14:05:50.985448 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:50.985473 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:50.986537 (+ 1064us) server_negotiation.cc:407] Connection header received
0502 14:05:50.986887 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:50.986914 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:50.987178 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:50.987398 (+ 220us) server_negotiation.cc:226] Negotiated authn=JWT
0502 14:05:50.988311 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.989526 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:50.991230 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.991753 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:50.991991 (+ 238us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0502 14:05:51.029911 (+ 37920us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0502 14:05:51.030434 (+ 523us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":98}
W20240502 14:05:51.032128 1319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240502 14:05:51.032477 1345 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:05:50.984589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:46269 (local address 127.0.0.1:37344)
0502 14:05:50.986388 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:50.986423 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:50.986638 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:50.987395 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:50.987414 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:50.987446 (+ 32us) client_negotiation.cc:189] Negotiated authn=JWT
0502 14:05:50.988077 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:50.988108 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.989745 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:50.989778 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:50.990992 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:50.991028 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:50.991269 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:50.991345 (+ 76us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0502 14:05:51.030294 (+ 38949us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0502 14:05:51.030706 (+ 412us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:46269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1234,"thread_start_us":871,"threads_started":1}
I20240502 14:05:51.034391 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1248
2024-05-02T14:05:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (1945 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-02T14:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:51Z Disabled control of system clock
I20240502 14:05:51.124899 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35795
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:35795
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/testchainca.cert with env {}
W20240502 14:05:51.418193 1350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:51.418727 1350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:51.419266 1350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:51.448032 1350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:51.448343 1350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:51.448503 1350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:51.448693 1350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:51.448876 1350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240502 14:05:51.449059 1350 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240502 14:05:51.449200 1350 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240502 14:05:51.482354 1350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:35795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:51.483449 1350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:51.484437 1350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:51.499888 1356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:51.499994 1359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:51.500603 1357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:51.502220 1350 server_base.cc:1034] running on GCE node
I20240502 14:05:52.543067 1350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:05:52.546226 1350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:05:52.547631 1350 hybrid_clock.cc:648] HybridClock initialized: now 1714658752547589 us; error 61 us; skew 500 ppm
I20240502 14:05:52.548381 1350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:52.555269 1350 webserver.cc:460] Webserver started at http://127.27.174.126:34293/ using document root <none> and password file <none>
I20240502 14:05:52.556135 1350 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:52.556336 1350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:52.556754 1350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:52.561913 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "08814e20821c4d6ca286d7d10d683ebd"
format_stamp: "Formatted at 2024-05-02 14:05:52 on dist-test-slave-dlb8"
server_key: "55f4ce1e7668bdf623125db2830380ee"
server_key_iv: "907c231b4e5fd4cee0521f579598e3f8"
server_key_version: "encryptionkey@0"
I20240502 14:05:52.563068 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "08814e20821c4d6ca286d7d10d683ebd"
format_stamp: "Formatted at 2024-05-02 14:05:52 on dist-test-slave-dlb8"
server_key: "55f4ce1e7668bdf623125db2830380ee"
server_key_iv: "907c231b4e5fd4cee0521f579598e3f8"
server_key_version: "encryptionkey@0"
I20240502 14:05:52.570005 1350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:05:52.575430 1366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:52.576447 1350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:05:52.576747 1350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "08814e20821c4d6ca286d7d10d683ebd"
format_stamp: "Formatted at 2024-05-02 14:05:52 on dist-test-slave-dlb8"
server_key: "55f4ce1e7668bdf623125db2830380ee"
server_key_iv: "907c231b4e5fd4cee0521f579598e3f8"
server_key_version: "encryptionkey@0"
I20240502 14:05:52.577108 1350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:52.642269 1350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:52.643823 1350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:05:52.644306 1350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:52.713606 1350 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:35795
I20240502 14:05:52.713696 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:35795 every 8 connection(s)
I20240502 14:05:52.716310 1350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:05:52.720988 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1350
I20240502 14:05:52.721205 1418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:52.721431 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:05:52.722608 28345 external_mini_cluster.cc:1428] Setting key 7fdee4345c4297dc09387798a929aac4
I20240502 14:05:52.743420 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:52.724878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.27.174.126:35795)
0502 14:05:52.727770 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:52.727794 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:52.727847 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:05:52.732754 (+ 4907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:52.732799 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:52.733208 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:52.733547 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:05:52.735337 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.736747 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:52.738495 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.738915 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:52.740016 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:05:52.740110 (+ 94us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:05:52.740141 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:05:52.740876 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:05:52.740970 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:05:52.741009 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:05:52.741130 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0502 14:05:52.741491 (+ 361us) server_negotiation.cc:299] Negotiation successful
0502 14:05:52.741752 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":1821,"threads_started":1}
I20240502 14:05:52.746456 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd: Bootstrap starting.
I20240502 14:05:52.747884 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:05:52.751564 1418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:52.753300 1418 log.cc:824] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd: Log is configured to *not* fsync() on all Append() calls
I20240502 14:05:52.758204 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd: No bootstrap required, opened a new log
I20240502 14:05:52.777439 1418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } }
I20240502 14:05:52.778134 1418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:52.778367 1418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08814e20821c4d6ca286d7d10d683ebd, State: Initialized, Role: FOLLOWER
I20240502 14:05:52.779091 1418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } }
I20240502 14:05:52.779759 1418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:52.780030 1418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:52.780330 1418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:52.784214 1418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } }
I20240502 14:05:52.784879 1418 leader_election.cc:302] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08814e20821c4d6ca286d7d10d683ebd; no voters:
I20240502 14:05:52.786631 1418 leader_election.cc:288] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:52.786962 1423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:52.789463 1423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [term 1 LEADER]: Becoming Leader. State: Replica: 08814e20821c4d6ca286d7d10d683ebd, State: Running, Role: LEADER
I20240502 14:05:52.790197 1423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } }
I20240502 14:05:52.790989 1418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:52.803215 1425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08814e20821c4d6ca286d7d10d683ebd. Latest consensus state: current_term: 1 leader_uuid: "08814e20821c4d6ca286d7d10d683ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } } }
I20240502 14:05:52.804108 1425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:52.806349 1424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08814e20821c4d6ca286d7d10d683ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08814e20821c4d6ca286d7d10d683ebd" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 35795 } } }
I20240502 14:05:52.807137 1424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:52.820508 1432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:52.831205 1432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:52.845975 1432 catalog_manager.cc:1347] Generated new cluster ID: 8b6cfa844a7847bc9d9a3243c3414faa
I20240502 14:05:52.846304 1432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:52.903870 1432 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:52.905910 1432 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:52.922111 1432 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 08814e20821c4d6ca286d7d10d683ebd: Generated new TSK 0
I20240502 14:05:52.923205 1432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:53.014753 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:05:52.962334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.27.174.126:35795)
0502 14:05:52.963235 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0502 14:05:52.963257 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:05:52.964034 (+ 777us) server_negotiation.cc:407] Connection header received
0502 14:05:52.964509 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:05:52.964527 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:05:52.964727 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:05:52.964881 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0502 14:05:52.966205 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.966986 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:05:52.968873 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.969222 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:52.969410 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0502 14:05:53.013355 (+ 43945us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0502 14:05:53.013808 (+ 453us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37280: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":114}
W20240502 14:05:53.014988 1421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37280: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20240502 14:05:53.016062 1447 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:05:52.961964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:35795 (local address 127.0.0.1:37280)
0502 14:05:52.963850 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0502 14:05:52.963905 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0502 14:05:52.964249 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:05:52.964950 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:05:52.964984 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:05:52.965040 (+ 56us) client_negotiation.cc:189] Negotiated authn=JWT
0502 14:05:52.965922 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:52.965973 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.967207 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:05:52.967241 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:05:52.968610 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:05:52.968659 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:05:52.968987 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:05:52.969093 (+ 106us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0502 14:05:53.013796 (+ 44703us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0502 14:05:53.014284 (+ 488us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.174.126:35795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":1181,"thread_start_us":796,"threads_started":1}
I20240502 14:05:53.017725 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1350
2024-05-02T14:05:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1967 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (2262 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (2280 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:05:57 dist-test-slave-dlb8 krb5kdc[1464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:05:57 dist-test-slave-dlb8 krb5kdc[1464](info): set up 2 sockets
May 02 14:05:57 dist-test-slave-dlb8 krb5kdc[1464](info): commencing operation
krb5kdc: starting...
W20240502 14:05:58.547973 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:05:58 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:05:58.758298 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36309
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:36309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:05:59.038594 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:05:59.039191 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:05:59.039731 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:05:59.068339 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:05:59.068734 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:05:59.069020 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:05:59.069253 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:05:59.069490 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:05:59.103459 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:36309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:36309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:05:59.104689 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:05:59.105710 1477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:59.120404 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:59.121120 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:59.122598 1477 server_base.cc:1034] running on GCE node
W20240502 14:05:59.121717 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.170553 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:00.172930 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:00.174237 1477 hybrid_clock.cc:648] HybridClock initialized: now 1714658760174201 us; error 53 us; skew 500 ppm
I20240502 14:06:00.175020 1477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:00 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658760, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:00.180526 1477 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:06:00.184814 1477 webserver.cc:460] Webserver started at http://127.27.174.126:36637/ using document root <none> and password file <none>
I20240502 14:06:00.185710 1477 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:00.185920 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:00.186367 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:00.191103 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "dd6025fb91c948c8bbe6ca79bf7a6891"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-dlb8"
server_key: "06b61f03484ac122e1d3a5351231913f"
server_key_iv: "a93efea5aca00ee14194c963eb212e4c"
server_key_version: "encryptionkey@0"
I20240502 14:06:00.192171 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "dd6025fb91c948c8bbe6ca79bf7a6891"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-dlb8"
server_key: "06b61f03484ac122e1d3a5351231913f"
server_key_iv: "a93efea5aca00ee14194c963eb212e4c"
server_key_version: "encryptionkey@0"
I20240502 14:06:00.198698 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:06:00.203624 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.204794 1477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:06:00.205116 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "dd6025fb91c948c8bbe6ca79bf7a6891"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-dlb8"
server_key: "06b61f03484ac122e1d3a5351231913f"
server_key_iv: "a93efea5aca00ee14194c963eb212e4c"
server_key_version: "encryptionkey@0"
I20240502 14:06:00.205490 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:00.252616 1477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:00.257969 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:00.258468 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:00.322660 1477 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:36309
I20240502 14:06:00.322736 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:36309 every 8 connection(s)
I20240502 14:06:00.325254 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:06:00.330154 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:00.335723 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1477
I20240502 14:06:00.336277 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:06:00.337479 28345 external_mini_cluster.cc:1428] Setting key 2c9c35296260eb08cbf98f1f381bbb15
I20240502 14:06:00.354110 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891: Bootstrap starting.
May 02 14:06:00 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:00.359683 1546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.361928 1546 log.cc:824] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:00.367973 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891: No bootstrap required, opened a new log
I20240502 14:06:00.371820 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:00.339756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.27.174.126:36309)
0502 14:06:00.342302 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:00.342337 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:00.342406 (+ 69us) server_negotiation.cc:407] Connection header received
0502 14:06:00.346705 (+ 4299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:00.346758 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:00.347231 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:00.347660 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:00.350022 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:00.352318 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:00.353653 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:00.354270 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:00.358981 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:00.359117 (+ 136us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:00.359154 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:00.359293 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:00.364683 (+ 5390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:00.366087 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:00.366117 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:00.366159 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:00.366392 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:00.366909 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:00.366940 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:00.366962 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:00.367735 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:00.368161 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:00.368716 (+ 555us) server_negotiation.cc:299] Negotiation successful
0502 14:06:00.369271 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":1461,"threads_started":1}
I20240502 14:06:00.388696 1546 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } }
I20240502 14:06:00.389647 1546 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.389842 1546 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6025fb91c948c8bbe6ca79bf7a6891, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.390511 1546 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } }
I20240502 14:06:00.391037 1546 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:00.391367 1546 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:00.391709 1546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.395684 1546 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } }
I20240502 14:06:00.396358 1546 leader_election.cc:302] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6025fb91c948c8bbe6ca79bf7a6891; no voters:
I20240502 14:06:00.397888 1546 leader_election.cc:288] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:00.398324 1551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.400918 1551 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [term 1 LEADER]: Becoming Leader. State: Replica: dd6025fb91c948c8bbe6ca79bf7a6891, State: Running, Role: LEADER
I20240502 14:06:00.401929 1546 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:00.401778 1551 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } }
I20240502 14:06:00.412649 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6025fb91c948c8bbe6ca79bf7a6891. Latest consensus state: current_term: 1 leader_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } } }
I20240502 14:06:00.413472 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:00.415773 1552 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6025fb91c948c8bbe6ca79bf7a6891" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 36309 } } }
I20240502 14:06:00.416400 1552 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:00.418475 1561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:00.429742 1561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:00.445420 1561 catalog_manager.cc:1347] Generated new cluster ID: 3f7a226479e34d579f6452ba6cc131d6
I20240502 14:06:00.445742 1561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:00.479830 1561 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:00.481849 1561 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:00.496088 1561 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dd6025fb91c948c8bbe6ca79bf7a6891: Generated new TSK 0
I20240502 14:06:00.497134 1561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:06:00.558490 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:00.853021 1572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:00.853528 1572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:00.854123 1572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:00.882891 1572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:00.883261 1572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:00.884124 1572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:06:00.917543 1572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:00.918717 1572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:00.919718 1572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:00.937376 1579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.939183 1572 server_base.cc:1034] running on GCE node
W20240502 14:06:00.940593 1581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.939380 1578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:01.988032 1572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:01.990823 1572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:01.992234 1572 hybrid_clock.cc:648] HybridClock initialized: now 1714658761992178 us; error 77 us; skew 500 ppm
I20240502 14:06:01.992987 1572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:01 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658761, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:01.998577 1572 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:06:02.004665 1572 webserver.cc:460] Webserver started at http://127.27.174.65:41303/ using document root <none> and password file <none>
I20240502 14:06:02.005553 1572 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:02.005741 1572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:02.006237 1572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:02.011184 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "9f293aaa92e742669bbe81665900735e"
format_stamp: "Formatted at 2024-05-02 14:06:02 on dist-test-slave-dlb8"
server_key: "32efce35866da726f2208c457baa7260"
server_key_iv: "c329eeef69f9faad41717bd8b40430c4"
server_key_version: "encryptionkey@0"
I20240502 14:06:02.012291 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "9f293aaa92e742669bbe81665900735e"
format_stamp: "Formatted at 2024-05-02 14:06:02 on dist-test-slave-dlb8"
server_key: "32efce35866da726f2208c457baa7260"
server_key_iv: "c329eeef69f9faad41717bd8b40430c4"
server_key_version: "encryptionkey@0"
I20240502 14:06:02.019819 1572 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240502 14:06:02.025579 1589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.026664 1572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:06:02.026954 1572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "9f293aaa92e742669bbe81665900735e"
format_stamp: "Formatted at 2024-05-02 14:06:02 on dist-test-slave-dlb8"
server_key: "32efce35866da726f2208c457baa7260"
server_key_iv: "c329eeef69f9faad41717bd8b40430c4"
server_key_version: "encryptionkey@0"
I20240502 14:06:02.027297 1572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:02.085804 1572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:02.091488 1572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:02.091897 1572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:02.094713 1572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:02.098577 1572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:02.098754 1572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.099004 1572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:02.099166 1572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.222672 1572 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35725
I20240502 14:06:02.222769 1701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35725 every 8 connection(s)
I20240502 14:06:02.225082 1572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:06:02.233146 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1572
I20240502 14:06:02.233556 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:06:02.234586 28345 external_mini_cluster.cc:1428] Setting key 18c5e41fac478d0cd80aa66f5180584a
May 02 14:06:02 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658761, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20240502 14:06:02.255550 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:02.228837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:52355 (local address 127.27.174.126:36309)
0502 14:06:02.230300 (+ 1463us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:02.230331 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:02.231530 (+ 1199us) server_negotiation.cc:407] Connection header received
0502 14:06:02.236067 (+ 4537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:02.236096 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:02.236319 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:02.236523 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:02.239234 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:02.240016 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:02.241318 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:02.241866 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:02.246113 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:02.246198 (+ 85us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:02.246216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:02.246317 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:02.249707 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:02.250768 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:02.250791 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:02.250804 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:02.250923 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:02.251472 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:02.251492 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:02.251505 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:02.251844 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:02.252047 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:02.252957 (+ 910us) server_negotiation.cc:299] Negotiation successful
0502 14:06:02.253382 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":454,"threads_started":1}
I20240502 14:06:02.256119 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:02.229045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36309 (local address 127.27.174.65:52355)
0502 14:06:02.231298 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:02.231362 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:02.235665 (+ 4303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:02.236798 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:02.236835 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:02.237930 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:02.238980 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:02.239016 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:02.240241 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:02.240262 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:02.241088 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:02.241121 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:02.241487 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:02.242508 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:02.242572 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:02.245826 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:02.249903 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:02.249929 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:02.249950 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:02.250586 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:02.251077 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:02.251097 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:02.251110 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:02.251298 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:02.252027 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:02.252055 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:02.252434 (+ 379us) client_negotiation.cc:769] Sending connection context
0502 14:06:02.252740 (+ 306us) client_negotiation.cc:240] Negotiation successful
0502 14:06:02.253307 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":1324,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:06:02.258235 1702 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36309
I20240502 14:06:02.258688 1702 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:02.260011 1702 heartbeater.cc:511] Master 127.27.174.126:36309 requested a full tablet report, sending...
I20240502 14:06:02.262801 1511 ts_manager.cc:194] Registered new tserver with Master: 9f293aaa92e742669bbe81665900735e (127.27.174.65:35725)
I20240502 14:06:02.264456 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:52355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:06:02.282686 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:02.565850 1708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:02.566391 1708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:02.566944 1708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:02.594837 1708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:02.595233 1708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:02.596125 1708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:06:02.630810 1708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:02.631959 1708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:02.632984 1708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:02.648146 1715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:03.267617 1702 heartbeater.cc:503] Master 127.27.174.126:36309 was elected leader, sending a full tablet report...
W20240502 14:06:02.648447 1714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:02.650022 1717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:03.752065 1716 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1101 milliseconds
I20240502 14:06:03.752173 1708 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:06:03.753232 1708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:03.755507 1708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:03.756850 1708 hybrid_clock.cc:648] HybridClock initialized: now 1714658763756824 us; error 41 us; skew 500 ppm
I20240502 14:06:03.757694 1708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:03 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:03.763540 1708 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:06:03.767866 1708 webserver.cc:460] Webserver started at http://127.27.174.66:45307/ using document root <none> and password file <none>
I20240502 14:06:03.768672 1708 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:03.768877 1708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:03.769256 1708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:03.773823 1708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "101b328029dc4e0f8aba58ac7bafd045"
format_stamp: "Formatted at 2024-05-02 14:06:03 on dist-test-slave-dlb8"
server_key: "fd0d5ab5219b8fe5b6f73229bc973188"
server_key_iv: "87d92980c3d5fcd6ddfd1398b7b0fa42"
server_key_version: "encryptionkey@0"
I20240502 14:06:03.775022 1708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "101b328029dc4e0f8aba58ac7bafd045"
format_stamp: "Formatted at 2024-05-02 14:06:03 on dist-test-slave-dlb8"
server_key: "fd0d5ab5219b8fe5b6f73229bc973188"
server_key_iv: "87d92980c3d5fcd6ddfd1398b7b0fa42"
server_key_version: "encryptionkey@0"
I20240502 14:06:03.781831 1708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:06:03.786890 1725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:03.787863 1708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:06:03.788156 1708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "101b328029dc4e0f8aba58ac7bafd045"
format_stamp: "Formatted at 2024-05-02 14:06:03 on dist-test-slave-dlb8"
server_key: "fd0d5ab5219b8fe5b6f73229bc973188"
server_key_iv: "87d92980c3d5fcd6ddfd1398b7b0fa42"
server_key_version: "encryptionkey@0"
I20240502 14:06:03.788496 1708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:03.838968 1708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:03.843976 1708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:03.844420 1708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:03.846874 1708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:03.850826 1708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:03.851032 1708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:03.851270 1708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:03.851420 1708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:03.977840 1708 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:34971
I20240502 14:06:03.977949 1837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:34971 every 8 connection(s)
I20240502 14:06:03.980281 1708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:06:03.987078 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1708
I20240502 14:06:03.987542 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:06:03.988536 28345 external_mini_cluster.cc:1428] Setting key d727709f0bb1a5cf9cdd180396bd1ba2
May 02 14:06:03 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:06:04.011013 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:03.984168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39175 (local address 127.27.174.126:36309)
0502 14:06:03.985203 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:03.985216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:03.987696 (+ 2480us) server_negotiation.cc:407] Connection header received
0502 14:06:03.992473 (+ 4777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:03.992503 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:03.992717 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:03.992903 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:03.995052 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:03.996439 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:03.997699 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:03.998269 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:04.001578 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:04.001649 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:04.001666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:04.001773 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:04.005254 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:04.006425 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:04.006449 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:04.006472 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:04.006597 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:04.007245 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:04.007279 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:04.007305 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:04.007846 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:04.008127 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:04.009118 (+ 991us) server_negotiation.cc:299] Negotiation successful
0502 14:06:04.009575 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":418,"threads_started":1}
I20240502 14:06:04.011409 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:03.984524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36309 (local address 127.27.174.66:39175)
0502 14:06:03.987477 (+ 2953us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:03.987542 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:03.992144 (+ 4602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:03.993075 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:03.993098 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:03.993834 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:03.994785 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:03.994823 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:03.996669 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:03.996691 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:03.997487 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:03.997518 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:03.997781 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:03.998580 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:03.998640 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:04.001321 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:04.005463 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:04.005493 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:04.005516 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:04.006226 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:04.006764 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:04.006782 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:04.006796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:04.006980 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:04.008039 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:04.008060 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:04.008417 (+ 357us) client_negotiation.cc:769] Sending connection context
0502 14:06:04.008864 (+ 447us) client_negotiation.cc:240] Negotiation successful
0502 14:06:04.009451 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"thread_start_us":1673,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:06:04.013212 1838 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36309
I20240502 14:06:04.013576 1838 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:04.014608 1838 heartbeater.cc:511] Master 127.27.174.126:36309 requested a full tablet report, sending...
I20240502 14:06:04.016713 1511 ts_manager.cc:194] Registered new tserver with Master: 101b328029dc4e0f8aba58ac7bafd045 (127.27.174.66:34971)
I20240502 14:06:04.017860 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:39175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:06:04.036808 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:04.333304 1844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:04.333985 1844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:04.335206 1844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:04.370450 1844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:04.370745 1844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:04.371532 1844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:06:04.404814 1844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:36309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:04.405997 1844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:04.406999 1844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:04.424023 1851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:05.021042 1838 heartbeater.cc:503] Master 127.27.174.126:36309 was elected leader, sending a full tablet report...
W20240502 14:06:04.424012 1850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:04.425005 1853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:04.425899 1844 server_base.cc:1034] running on GCE node
I20240502 14:06:05.483888 1844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:05.486194 1844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:05.487550 1844 hybrid_clock.cc:648] HybridClock initialized: now 1714658765487512 us; error 59 us; skew 500 ppm
I20240502 14:06:05.488265 1844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.493854 1844 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:06:05.498272 1844 webserver.cc:460] Webserver started at http://127.27.174.67:38347/ using document root <none> and password file <none>
I20240502 14:06:05.499068 1844 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:05.499238 1844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:05.499600 1844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:05.504345 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "ae6e39eba0e140f69c8440682a6ada62"
format_stamp: "Formatted at 2024-05-02 14:06:05 on dist-test-slave-dlb8"
server_key: "7cee325172231030c5c5d2f5137b9894"
server_key_iv: "7d606e68f348a6d453b5d95a0a319c9c"
server_key_version: "encryptionkey@0"
I20240502 14:06:05.505427 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "ae6e39eba0e140f69c8440682a6ada62"
format_stamp: "Formatted at 2024-05-02 14:06:05 on dist-test-slave-dlb8"
server_key: "7cee325172231030c5c5d2f5137b9894"
server_key_iv: "7d606e68f348a6d453b5d95a0a319c9c"
server_key_version: "encryptionkey@0"
I20240502 14:06:05.512138 1844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20240502 14:06:05.517529 1861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:05.518482 1844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:06:05.518780 1844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "ae6e39eba0e140f69c8440682a6ada62"
format_stamp: "Formatted at 2024-05-02 14:06:05 on dist-test-slave-dlb8"
server_key: "7cee325172231030c5c5d2f5137b9894"
server_key_iv: "7d606e68f348a6d453b5d95a0a319c9c"
server_key_version: "encryptionkey@0"
I20240502 14:06:05.519089 1844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:05.575927 1844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:05.580942 1844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:05.581385 1844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:05.583745 1844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:05.587378 1844 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:05.587549 1844 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:05.587813 1844 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:05.587970 1844 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:05.717694 1844 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:38277
I20240502 14:06:05.717782 1973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:38277 every 8 connection(s)
I20240502 14:06:05.720175 1844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:06:05.721917 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 1844
I20240502 14:06:05.722421 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:06:05.723623 28345 external_mini_cluster.cc:1428] Setting key 56c4187b58093a1aefeff8df3951b2be
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:05.752683 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:05.725259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:54497 (local address 127.27.174.126:36309)
0502 14:06:05.726981 (+ 1722us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.727006 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.728481 (+ 1475us) server_negotiation.cc:407] Connection header received
0502 14:06:05.733160 (+ 4679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.733180 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.733344 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.733487 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.735830 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.736939 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:05.738230 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.738638 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:05.743042 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:05.743114 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:05.743130 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:05.743229 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:05.746636 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.747697 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.747720 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.747734 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.747871 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.748450 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.748475 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.748488 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.748880 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:05.749099 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:05.750020 (+ 921us) server_negotiation.cc:299] Negotiation successful
0502 14:06:05.750494 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":498,"threads_started":1}
I20240502 14:06:05.753188 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:05.725630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:36309 (local address 127.27.174.67:54497)
0502 14:06:05.728281 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:05.728344 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:05.732827 (+ 4483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:05.733696 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:05.733722 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:05.734543 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:05.735597 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:05.735640 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.737158 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:05.737182 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:05.737986 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:05.738024 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.738415 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:05.739233 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:05.739297 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:05.742758 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:05.746836 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:05.746860 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:05.746882 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:05.747505 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:05.748032 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:05.748055 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:05.748070 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:05.748265 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:05.749075 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:05.749106 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:05.749492 (+ 386us) client_negotiation.cc:769] Sending connection context
0502 14:06:05.749808 (+ 316us) client_negotiation.cc:240] Negotiation successful
0502 14:06:05.750448 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"thread_start_us":1492,"threads_started":1}
I20240502 14:06:05.755421 1974 heartbeater.cc:347] Connected to a master server at 127.27.174.126:36309
I20240502 14:06:05.755831 1974 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:05.756942 1974 heartbeater.cc:511] Master 127.27.174.126:36309 requested a full tablet report, sending...
I20240502 14:06:05.758922 1511 ts_manager.cc:194] Registered new tserver with Master: ae6e39eba0e140f69c8440682a6ada62 (127.27.174.67:38277)
I20240502 14:06:05.760169 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:54497
I20240502 14:06:05.771160 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/test-admin.keytab.
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.802884 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240502 14:06:05.812407 28345 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc' of type FILE
W20240502 14:06:05.812799 1982 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:05.807894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.27.174.65:35725)
0502 14:06:05.809641 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.809663 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.809705 (+ 42us) server_negotiation.cc:407] Connection header received
0502 14:06:05.809926 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.809945 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.810237 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.810402 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.811655 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39652: BlockingRecv error: recv got EOF from 127.0.0.1:39652 (error 108)
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1013,"threads_started":1}
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.814561 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.816303 28345 init.cc:295] Successfully reacquired a new kerberos TGT
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:06:05.838945 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:05.819328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.27.174.65:35725)
0502 14:06:05.820014 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.820045 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.820577 (+ 532us) server_negotiation.cc:407] Connection header received
0502 14:06:05.820936 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.820954 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.821109 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.821260 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.823195 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.824460 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:05.825732 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.826128 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:05.829956 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:05.830085 (+ 129us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:05.830119 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:05.830207 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:05.833701 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.835024 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.835048 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.835070 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.835261 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.835841 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.835861 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.835878 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.836302 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:05.836664 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:05.837137 (+ 473us) server_negotiation.cc:299] Negotiation successful
0502 14:06:05.837546 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20240502 14:06:05.848145 28345 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc)
W20240502 14:06:05.848351 1982 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:05.844972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.27.174.65:35725)
0502 14:06:05.845667 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.845685 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.845948 (+ 263us) server_negotiation.cc:407] Connection header received
0502 14:06:05.846264 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.846284 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.846439 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.846568 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.847508 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39674: BlockingRecv error: recv got EOF from 127.0.0.1:39674 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.849977 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.851485 28345 init.cc:295] Successfully reacquired a new kerberos TGT
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:06:05.870842 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:05.854216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.27.174.65:35725)
0502 14:06:05.854854 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.854883 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.855390 (+ 507us) server_negotiation.cc:407] Connection header received
0502 14:06:05.855724 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.855742 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.855901 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.856045 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.857626 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.858522 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:05.859779 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.860280 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:05.863666 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:05.863737 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:05.863754 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:05.863846 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:05.866892 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.867743 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.867761 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.867775 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.867905 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.868312 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.868332 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.868344 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.868669 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:05.868897 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:05.869360 (+ 463us) server_negotiation.cc:299] Negotiation successful
0502 14:06:05.869655 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20240502 14:06:05.879137 28345 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714658657003751-28345-0/krb5kdc/krb5cc' of type FILE
W20240502 14:06:05.879423 1982 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:05.876094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.27.174.65:35725)
0502 14:06:05.876800 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.876818 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.877106 (+ 288us) server_negotiation.cc:407] Connection header received
0502 14:06:05.877358 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.877380 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.877530 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.877653 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.878598 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39684: BlockingRecv error: recv got EOF from 127.0.0.1:39684 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.881059 28345 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:05.882512 28345 init.cc:295] Successfully reacquired a new kerberos TGT
May 02 14:06:05 dist-test-slave-dlb8 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.65@KRBTEST.COM
I20240502 14:06:05.901218 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:05.885044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.27.174.65:35725)
0502 14:06:05.885772 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:05.885790 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:05.886061 (+ 271us) server_negotiation.cc:407] Connection header received
0502 14:06:05.886349 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:05.886366 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:05.886524 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:05.886664 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:05.888262 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.888990 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:05.890077 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:05.890391 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:05.893914 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:05.893982 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:05.893998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:05.894115 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:05.897097 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.897957 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.897976 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.897989 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.898153 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:05.898642 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:05.898661 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:05.898674 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:05.899014 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:05.899258 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:05.899689 (+ 431us) server_negotiation.cc:299] Negotiation successful
0502 14:06:05.899972 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20240502 14:06:05.903683 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1572
I20240502 14:06:05.926331 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1708
I20240502 14:06:05.950289 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1844
I20240502 14:06:05.971047 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 1477
2024-05-02T14:06:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (8417 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:877: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:06:06 dist-test-slave-dlb8 krb5kdc[1995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:06:06 dist-test-slave-dlb8 krb5kdc[1995](info): set up 2 sockets
May 02 14:06:06 dist-test-slave-dlb8 krb5kdc[1995](info): commencing operation
krb5kdc: starting...
W20240502 14:06:06.947818 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:06:07 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:06:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
Entry for principal oryx/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.126.keytab.
I20240502 14:06:07.151077 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41651
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:07.434096 2008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:07.434687 2008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:07.435249 2008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:07.464385 2008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:07.464744 2008 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240502 14:06:07.465032 2008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:07.465310 2008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:07.465564 2008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:07.465797 2008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:07.498554 2008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41651
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.126.keytab
--principal=oryx/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:07.499663 2008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:07.500687 2008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:07.515080 2015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:07.515164 2016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:07.517223 2008 server_base.cc:1034] running on GCE node
W20240502 14:06:07.516006 2018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:08.560448 2008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:08.563295 2008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:08.564718 2008 hybrid_clock.cc:648] HybridClock initialized: now 1714658768564662 us; error 78 us; skew 500 ppm
I20240502 14:06:08.565470 2008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:08 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658768, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:08.576210 2008 init.cc:377] Logged in from keytab as oryx/127.27.174.126@KRBTEST.COM (short username oryx)
I20240502 14:06:08.581130 2008 webserver.cc:460] Webserver started at http://127.27.174.126:36035/ using document root <none> and password file <none>
I20240502 14:06:08.581965 2008 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:08.582193 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:08.582579 2008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:08.587498 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "092ba44513f54368a9d741ec25b93335"
format_stamp: "Formatted at 2024-05-02 14:06:08 on dist-test-slave-dlb8"
server_key: "7b50869ab783ccc9f902e9bb3c9e82a0"
server_key_iv: "bb7ec7e718f0b8381dcbe705ed6cf30b"
server_key_version: "encryptionkey@0"
I20240502 14:06:08.588534 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "092ba44513f54368a9d741ec25b93335"
format_stamp: "Formatted at 2024-05-02 14:06:08 on dist-test-slave-dlb8"
server_key: "7b50869ab783ccc9f902e9bb3c9e82a0"
server_key_iv: "bb7ec7e718f0b8381dcbe705ed6cf30b"
server_key_version: "encryptionkey@0"
I20240502 14:06:08.595484 2008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:06:08.600688 2026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:08.601651 2008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:06:08.601941 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "092ba44513f54368a9d741ec25b93335"
format_stamp: "Formatted at 2024-05-02 14:06:08 on dist-test-slave-dlb8"
server_key: "7b50869ab783ccc9f902e9bb3c9e82a0"
server_key_iv: "bb7ec7e718f0b8381dcbe705ed6cf30b"
server_key_version: "encryptionkey@0"
I20240502 14:06:08.602319 2008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:08.656920 2008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:08.662353 2008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:08.662766 2008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:08.744527 2008 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41651
I20240502 14:06:08.744624 2087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41651 every 8 connection(s)
I20240502 14:06:08.747254 2008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:06:08.752171 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:08.757352 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2008
I20240502 14:06:08.757989 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:06:08.759186 28345 external_mini_cluster.cc:1428] Setting key 517aacb09da9e6e3d328c39116b4a88a
I20240502 14:06:08.770931 2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Bootstrap starting.
I20240502 14:06:08.777312 2088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:08.779577 2088 log.cc:824] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Log is configured to *not* fsync() on all Append() calls
May 02 14:06:08 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:08.785115 2088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: No bootstrap required, opened a new log
I20240502 14:06:08.795039 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:08.761965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.27.174.126:41651)
0502 14:06:08.764743 (+ 2778us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:08.764770 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:08.764821 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:06:08.770397 (+ 5576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:08.770437 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:08.770792 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:08.771135 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:08.774960 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:08.776348 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:08.778276 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:08.778698 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:08.784919 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:08.785014 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:08.785043 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:08.785142 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:08.789061 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:08.790405 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:08.790427 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:08.790461 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:08.790624 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:08.791193 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:08.791209 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:08.791221 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:08.791729 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:08.792035 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:08.792698 (+ 663us) server_negotiation.cc:299] Negotiation successful
0502 14:06:08.793094 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2241,"thread_start_us":1751,"threads_started":1}
I20240502 14:06:08.805907 2088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } }
I20240502 14:06:08.806653 2088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:08.806917 2088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092ba44513f54368a9d741ec25b93335, State: Initialized, Role: FOLLOWER
I20240502 14:06:08.807638 2088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } }
I20240502 14:06:08.808144 2088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:08.808449 2088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:08.808784 2088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:08.813064 2088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } }
I20240502 14:06:08.813793 2088 leader_election.cc:302] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092ba44513f54368a9d741ec25b93335; no voters:
I20240502 14:06:08.815459 2088 leader_election.cc:288] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:08.815835 2093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:08.818244 2093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [term 1 LEADER]: Becoming Leader. State: Replica: 092ba44513f54368a9d741ec25b93335, State: Running, Role: LEADER
I20240502 14:06:08.819039 2093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } }
I20240502 14:06:08.820346 2088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:08.829061 2094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092ba44513f54368a9d741ec25b93335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } } }
I20240502 14:06:08.829066 2095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092ba44513f54368a9d741ec25b93335. Latest consensus state: current_term: 1 leader_uuid: "092ba44513f54368a9d741ec25b93335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092ba44513f54368a9d741ec25b93335" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41651 } } }
I20240502 14:06:08.829973 2094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:08.830030 2095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:08.835445 2101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:08.844848 2101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:08.862793 2101 catalog_manager.cc:1347] Generated new cluster ID: 288e188a7bc44cc6a847f752c0adf12d
I20240502 14:06:08.863142 2101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:08.893129 2101 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:08.894860 2101 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:08.906548 2101 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Generated new TSK 0
I20240502 14:06:08.907421 2101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
I20240502 14:06:08.969604 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41651
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:09.259886 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:09.260432 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:09.261060 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:09.290756 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:09.291108 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:09.291726 2114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:06:09.292212 2114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:06:09.325232 2114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:09.326455 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:09.327456 2114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:09.344128 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:09.345758 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:09.348606 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:09.348351 2114 server_base.cc:1034] running on GCE node
I20240502 14:06:10.429796 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:10.432587 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:10.434010 2114 hybrid_clock.cc:648] HybridClock initialized: now 1714658770433959 us; error 75 us; skew 500 ppm
I20240502 14:06:10.434899 2114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:10 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658770, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:10.440836 2114 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20240502 14:06:10.446955 2114 webserver.cc:460] Webserver started at http://127.27.174.65:46093/ using document root <none> and password file <none>
I20240502 14:06:10.447829 2114 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:10.448048 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:10.448546 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:10.453543 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "97b50f777bb6437aafe6fb971346ed39"
format_stamp: "Formatted at 2024-05-02 14:06:10 on dist-test-slave-dlb8"
server_key: "9ef9a3e07ae197373555ed07f11f64f5"
server_key_iv: "be9bc03f3298d257e75c7fe1c711f6ad"
server_key_version: "encryptionkey@0"
I20240502 14:06:10.454672 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "97b50f777bb6437aafe6fb971346ed39"
format_stamp: "Formatted at 2024-05-02 14:06:10 on dist-test-slave-dlb8"
server_key: "9ef9a3e07ae197373555ed07f11f64f5"
server_key_iv: "be9bc03f3298d257e75c7fe1c711f6ad"
server_key_version: "encryptionkey@0"
I20240502 14:06:10.461877 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:06:10.467608 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:10.468571 2114 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:06:10.468847 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "97b50f777bb6437aafe6fb971346ed39"
format_stamp: "Formatted at 2024-05-02 14:06:10 on dist-test-slave-dlb8"
server_key: "9ef9a3e07ae197373555ed07f11f64f5"
server_key_iv: "be9bc03f3298d257e75c7fe1c711f6ad"
server_key_version: "encryptionkey@0"
I20240502 14:06:10.469146 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:10.523823 2114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:10.529114 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:10.529544 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:10.539897 2114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:10.540186 2114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:10.540550 2114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:10.540778 2114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:06:10 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658770, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:10.570834 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:10.535636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:54919 (local address 127.27.174.126:41651)
0502 14:06:10.538658 (+ 3022us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:10.538679 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:10.540317 (+ 1638us) server_negotiation.cc:407] Connection header received
0502 14:06:10.547958 (+ 7641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:10.547977 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:10.548129 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:10.548269 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:10.551566 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.552670 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:10.554645 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.555178 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:10.559152 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:10.559236 (+ 84us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:10.559253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:10.559347 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:10.564782 (+ 5435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:10.565883 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:10.565906 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:10.565920 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:10.566087 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:10.566701 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:10.566722 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:10.566736 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:10.567120 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:10.567339 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:10.568281 (+ 942us) server_negotiation.cc:299] Negotiation successful
0502 14:06:10.568723 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2257,"thread_start_us":1910,"threads_started":1}
I20240502 14:06:10.571367 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:10.535772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.65:54919)
0502 14:06:10.540062 (+ 4290us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:10.540148 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:10.547566 (+ 7418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:10.548647 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:10.548689 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:10.549825 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:10.551227 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:10.551294 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.553084 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:10.553119 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:10.554406 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:10.554460 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.554818 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:10.555899 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:10.555966 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:10.558893 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:10.565005 (+ 6112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:10.565027 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:10.565049 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:10.565672 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:10.566270 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:10.566289 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:10.566303 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:10.566506 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:10.567321 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:10.567351 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:10.567737 (+ 386us) client_negotiation.cc:769] Sending connection context
0502 14:06:10.568040 (+ 303us) client_negotiation.cc:240] Negotiation successful
0502 14:06:10.568674 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3566,"thread_start_us":2497,"threads_started":1}
I20240502 14:06:10.612668 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:10.585572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47437 (local address 127.27.174.126:41651)
0502 14:06:10.586459 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:10.586492 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:10.591746 (+ 5254us) server_negotiation.cc:407] Connection header received
0502 14:06:10.592072 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:10.592104 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:10.592336 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:10.592534 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:10.595473 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.596469 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:10.597899 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.598804 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:10.600379 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:10.600457 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:10.600474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:10.600569 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:10.605450 (+ 4881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:10.607012 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:10.607041 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:10.607066 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:10.607279 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:10.608078 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:10.608109 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:10.608137 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:10.608690 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:10.608983 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:10.609883 (+ 900us) server_negotiation.cc:299] Negotiation successful
0502 14:06:10.610453 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20240502 14:06:10.613265 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:10.585524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.65:47437)
0502 14:06:10.591593 (+ 6069us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:10.591624 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:10.591867 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:10.592624 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:10.592668 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:10.593883 (+ 1215us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:10.595137 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:10.595202 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.596718 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:10.596738 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:10.597688 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:10.597719 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:10.597998 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:10.598913 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:10.598994 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:10.600115 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:10.605718 (+ 5603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:10.605755 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:10.605776 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:10.606745 (+ 969us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:10.607522 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:10.607558 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:10.607593 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:10.607833 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:10.608910 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:10.608948 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:10.609298 (+ 350us) client_negotiation.cc:769] Sending connection context
0502 14:06:10.609650 (+ 352us) client_negotiation.cc:240] Negotiation successful
0502 14:06:10.610261 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5632,"thread_start_us":5037,"threads_started":1}
I20240502 14:06:10.694936 2114 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:32809
I20240502 14:06:10.695477 2251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:32809 every 8 connection(s)
I20240502 14:06:10.697297 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:06:10.701258 2252 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41651
I20240502 14:06:10.701643 2252 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:10.702625 2252 heartbeater.cc:511] Master 127.27.174.126:41651 requested a full tablet report, sending...
I20240502 14:06:10.704990 2043 ts_manager.cc:194] Registered new tserver with Master: 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809)
I20240502 14:06:10.706175 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2114
I20240502 14:06:10.706589 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:06:10.706910 2043 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:54919
I20240502 14:06:10.707666 28345 external_mini_cluster.cc:1428] Setting key b4d389ca50cbbd1d1f7fc72ddb354edf
WARNING: no policy specified for oryx/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
I20240502 14:06:10.756330 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41651
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:11.042734 2256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:11.043316 2256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:11.043905 2256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:11.072978 2256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:11.073313 2256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:11.073908 2256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:06:11.074445 2256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:06:11.107404 2256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:11.108462 2256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:11.109414 2256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:11.124444 2263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:11.709988 2252 heartbeater.cc:503] Master 127.27.174.126:41651 was elected leader, sending a full tablet report...
W20240502 14:06:11.125028 2265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:11.124454 2262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:11.126490 2256 server_base.cc:1034] running on GCE node
I20240502 14:06:12.171111 2256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:12.173357 2256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:12.174680 2256 hybrid_clock.cc:648] HybridClock initialized: now 1714658772174645 us; error 51 us; skew 500 ppm
I20240502 14:06:12.175398 2256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:12 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658772, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:12.180788 2256 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20240502 14:06:12.185670 2256 webserver.cc:460] Webserver started at http://127.27.174.66:43707/ using document root <none> and password file <none>
I20240502 14:06:12.186590 2256 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:12.186801 2256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:12.187240 2256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:12.192088 2256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "71e426200d0549cc8cab3b54553d69d0"
format_stamp: "Formatted at 2024-05-02 14:06:12 on dist-test-slave-dlb8"
server_key: "8efa627609965d6aeefedb0c53030a85"
server_key_iv: "3d727b41ca4106a9b74553ddd1df8466"
server_key_version: "encryptionkey@0"
I20240502 14:06:12.193156 2256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "71e426200d0549cc8cab3b54553d69d0"
format_stamp: "Formatted at 2024-05-02 14:06:12 on dist-test-slave-dlb8"
server_key: "8efa627609965d6aeefedb0c53030a85"
server_key_iv: "3d727b41ca4106a9b74553ddd1df8466"
server_key_version: "encryptionkey@0"
I20240502 14:06:12.199991 2256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20240502 14:06:12.205206 2273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:12.206209 2256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:06:12.206504 2256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "71e426200d0549cc8cab3b54553d69d0"
format_stamp: "Formatted at 2024-05-02 14:06:12 on dist-test-slave-dlb8"
server_key: "8efa627609965d6aeefedb0c53030a85"
server_key_iv: "3d727b41ca4106a9b74553ddd1df8466"
server_key_version: "encryptionkey@0"
I20240502 14:06:12.206864 2256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:12.266155 2256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:12.270866 2256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:12.271283 2256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:12.279564 2256 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:12.279784 2256 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:12.280005 2256 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:12.280144 2256 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:06:12 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658772, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:12.309098 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:12.277169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57409 (local address 127.27.174.126:41651)
0502 14:06:12.278966 (+ 1797us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:12.278995 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:12.281512 (+ 2517us) server_negotiation.cc:407] Connection header received
0502 14:06:12.288969 (+ 7457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:12.288988 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:12.289151 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:12.289309 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:12.293170 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.293992 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:12.295262 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.295793 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:12.299560 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:12.299629 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:12.299645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:12.299735 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:12.303311 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:12.304258 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:12.304281 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:12.304294 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:12.304412 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:12.304935 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:12.304953 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:12.304965 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:12.305363 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:12.305565 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:12.306559 (+ 994us) server_negotiation.cc:299] Negotiation successful
0502 14:06:12.306978 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":636,"threads_started":1}
I20240502 14:06:12.309619 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:12.277358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.66:57409)
0502 14:06:12.281204 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:12.281305 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:12.288464 (+ 7159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:12.289639 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:12.289680 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:12.290955 (+ 1275us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:12.292679 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:12.292783 (+ 104us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.294238 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:12.294257 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:12.295034 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:12.295069 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.295446 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:12.296406 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:12.296479 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:12.299328 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:12.303508 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:12.303533 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:12.303553 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:12.304083 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:12.304559 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:12.304579 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:12.304592 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:12.304768 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:12.305551 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:12.305579 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:12.305933 (+ 354us) client_negotiation.cc:769] Sending connection context
0502 14:06:12.306318 (+ 385us) client_negotiation.cc:240] Negotiation successful
0502 14:06:12.306947 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3025,"thread_start_us":2107,"threads_started":1}
I20240502 14:06:12.345710 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:12.325029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:40313 (local address 127.27.174.126:41651)
0502 14:06:12.325918 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:12.325971 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:12.328280 (+ 2309us) server_negotiation.cc:407] Connection header received
0502 14:06:12.328577 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:12.328595 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:12.328747 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:12.328878 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:12.330755 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.331526 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:12.333194 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.334061 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:12.335827 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:12.335921 (+ 94us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:12.335968 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:12.336085 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:12.340429 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:12.341345 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:12.341364 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:12.341379 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:12.341514 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:12.342060 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:12.342089 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:12.342104 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:12.342496 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:12.342707 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:12.343418 (+ 711us) server_negotiation.cc:299] Negotiation successful
0502 14:06:12.343815 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20240502 14:06:12.345989 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:12.324993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.66:40313)
0502 14:06:12.328138 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:12.328176 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:12.328370 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:12.328961 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:12.328988 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:12.329761 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:12.330521 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:12.330566 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.331803 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:12.331822 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:12.332979 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:12.333014 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:12.333297 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:12.334164 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:12.334241 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:12.335580 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:12.340613 (+ 5033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:12.340640 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:12.340658 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:12.341173 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:12.341663 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:12.341680 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:12.341693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:12.341872 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:12.342682 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:12.342713 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:12.342961 (+ 248us) client_negotiation.cc:769] Sending connection context
0502 14:06:12.343184 (+ 223us) client_negotiation.cc:240] Negotiation successful
0502 14:06:12.343638 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2697,"thread_start_us":2262,"threads_started":1}
I20240502 14:06:12.432368 2256 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41285
I20240502 14:06:12.432870 2393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41285 every 8 connection(s)
I20240502 14:06:12.434737 2256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:06:12.439218 2394 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41651
I20240502 14:06:12.439643 2394 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:12.440618 2394 heartbeater.cc:511] Master 127.27.174.126:41651 requested a full tablet report, sending...
I20240502 14:06:12.442113 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2256
I20240502 14:06:12.442534 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:06:12.442523 2043 ts_manager.cc:194] Registered new tserver with Master: 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:12.443542 28345 external_mini_cluster.cc:1428] Setting key a4d0485c23bc7740c4d4f126792920af
I20240502 14:06:12.443750 2043 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:57409
WARNING: no policy specified for oryx/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
I20240502 14:06:12.487556 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41651
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:12.808467 2398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:12.808959 2398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:12.809572 2398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:12.838793 2398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:12.839139 2398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:12.839694 2398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240502 14:06:12.840186 2398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:06:12.872278 2398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.174.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:12.873365 2398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:12.874329 2398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:12.888502 2404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:13.446748 2394 heartbeater.cc:503] Master 127.27.174.126:41651 was elected leader, sending a full tablet report...
W20240502 14:06:12.890322 2407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:12.888967 2405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:13.962527 2406 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1073 milliseconds
I20240502 14:06:13.962636 2398 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:06:13.963682 2398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:13.966464 2398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:13.967890 2398 hybrid_clock.cc:648] HybridClock initialized: now 1714658773967863 us; error 43 us; skew 500 ppm
I20240502 14:06:13.968634 2398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:13 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658773, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:13.979876 2398 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20240502 14:06:13.985289 2398 webserver.cc:460] Webserver started at http://127.27.174.67:35677/ using document root <none> and password file <none>
I20240502 14:06:13.986209 2398 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:13.986423 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:13.986850 2398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:13.991811 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "8da730156fda48e7a89bbe894e2c31f9"
format_stamp: "Formatted at 2024-05-02 14:06:13 on dist-test-slave-dlb8"
server_key: "8cc884809d6463ac93c423e19ef99bc8"
server_key_iv: "249132d17193b6440c6987929eedca49"
server_key_version: "encryptionkey@0"
I20240502 14:06:13.992877 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "8da730156fda48e7a89bbe894e2c31f9"
format_stamp: "Formatted at 2024-05-02 14:06:13 on dist-test-slave-dlb8"
server_key: "8cc884809d6463ac93c423e19ef99bc8"
server_key_iv: "249132d17193b6440c6987929eedca49"
server_key_version: "encryptionkey@0"
I20240502 14:06:13.999593 2398 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20240502 14:06:14.004907 2415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.005908 2398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:06:14.006232 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "8da730156fda48e7a89bbe894e2c31f9"
format_stamp: "Formatted at 2024-05-02 14:06:13 on dist-test-slave-dlb8"
server_key: "8cc884809d6463ac93c423e19ef99bc8"
server_key_iv: "249132d17193b6440c6987929eedca49"
server_key_version: "encryptionkey@0"
I20240502 14:06:14.006553 2398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:14.058693 2398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:14.064164 2398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:14.064548 2398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:14.073112 2398 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:14.073518 2398 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:06:14.073839 2398 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:14.074067 2398 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658773, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:14.101919 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.070481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:59719 (local address 127.27.174.126:41651)
0502 14:06:14.073082 (+ 2601us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.073104 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.074643 (+ 1539us) server_negotiation.cc:407] Connection header received
0502 14:06:14.079914 (+ 5271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.079947 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.080218 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.080427 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:14.082675 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.084327 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.085853 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.086510 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.090320 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:14.090420 (+ 100us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:14.090444 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:14.090576 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:14.094428 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.095949 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.095969 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.095983 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.096117 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.096712 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.096731 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.096745 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.097181 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:14.097404 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.098813 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.099352 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1750,"threads_started":1}
I20240502 14:06:14.102653 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.070684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.67:59719)
0502 14:06:14.074414 (+ 3730us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.074481 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.079543 (+ 5062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.080608 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.080634 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.081371 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:14.082348 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.082387 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.084616 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.084641 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.085600 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.085653 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.086001 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.087185 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:14.087254 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:14.090054 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:14.094634 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.094667 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.094712 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.095747 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.096300 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.096330 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.096346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.096539 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.097416 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:14.097452 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:14.098086 (+ 634us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.098535 (+ 449us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.099220 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3179,"thread_start_us":2643,"threads_started":1}
I20240502 14:06:14.144733 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.119044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:49795 (local address 127.27.174.126:41651)
0502 14:06:14.119750 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.119766 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.123219 (+ 3453us) server_negotiation.cc:407] Connection header received
0502 14:06:14.123715 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.123750 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.123978 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.124116 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:14.126390 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.127879 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.130104 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.130679 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.133526 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:14.133654 (+ 128us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:14.133680 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:14.133815 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:14.139310 (+ 5495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.140318 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.140343 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.140360 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.140495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.140979 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.140996 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.141008 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.141353 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:14.141561 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.142290 (+ 729us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.142733 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20240502 14:06:14.145046 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.118914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.27.174.67:49795)
0502 14:06:14.122970 (+ 4056us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.123039 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.123410 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.124201 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.124231 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.125044 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:14.126017 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.126102 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.128164 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.128200 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.129795 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.129842 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.130183 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.131244 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:14.131329 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:14.133162 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:14.139568 (+ 6406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.139589 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.139603 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.140133 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.140644 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.140661 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.140680 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.140817 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.141533 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:14.141558 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:14.141821 (+ 263us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.142074 (+ 253us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.142559 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3252,"thread_start_us":1853,"threads_started":1}
I20240502 14:06:14.217435 2398 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:44793
I20240502 14:06:14.217943 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:44793 every 8 connection(s)
I20240502 14:06:14.219732 2398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:06:14.223695 2536 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41651
I20240502 14:06:14.224188 2536 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:14.225077 2536 heartbeater.cc:511] Master 127.27.174.126:41651 requested a full tablet report, sending...
I20240502 14:06:14.227059 2043 ts_manager.cc:194] Registered new tserver with Master: 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:14.228240 2043 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:59719
I20240502 14:06:14.229880 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2398
I20240502 14:06:14.230372 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:06:14.231343 28345 external_mini_cluster.cc:1428] Setting key a6e2aeaab74e4986b9ee09cbb4d3b1e2
I20240502 14:06:14.235498 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:14.294703 2423 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:14.280313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.27.174.126:41651)
0502 14:06:14.280941 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.280969 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.281666 (+ 697us) server_negotiation.cc:407] Connection header received
0502 14:06:14.282068 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.282095 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.282255 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.282398 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:14.284618 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.285434 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.286879 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.287444 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.293468 (+ 6024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47402: BlockingRecv error: recv got EOF from 127.0.0.1:47402 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:14.318861 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.300831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.27.174.126:41651)
0502 14:06:14.301361 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.301385 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.302180 (+ 795us) server_negotiation.cc:407] Connection header received
0502 14:06:14.302515 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.302533 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.302689 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.302831 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:14.304640 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.305577 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.306918 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.307457 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.311083 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:14.311171 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:14.311188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:14.311290 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:14.314296 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.315158 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.315179 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.315192 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.315317 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.315775 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.315793 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.315806 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.316167 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:14.316534 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.317070 (+ 536us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.317528 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240502 14:06:14.324275 2043 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:14.343626 2043 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:14.381453 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.367764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.0.0.1:59332)
0502 14:06:14.371812 (+ 4048us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.371887 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.372507 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.373862 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.373901 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.374009 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:14.374768 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.374820 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.377361 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.377386 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.379194 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.379249 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.379552 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.379656 (+ 104us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.379803 (+ 147us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.380005 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3267,"thread_start_us":3045,"threads_started":1}
I20240502 14:06:14.383225 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.367905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.27.174.65:32809)
0502 14:06:14.370058 (+ 2153us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.370090 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.372035 (+ 1945us) server_negotiation.cc:407] Connection header received
0502 14:06:14.372868 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.372886 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.373109 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.373260 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:14.375057 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.377001 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.379450 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.380577 (+ 1127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.380804 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.380992 (+ 188us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.381221 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":1335,"threads_started":1}
I20240502 14:06:14.387825 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.370751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.27.174.66:41285)
0502 14:06:14.372780 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.372810 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.377590 (+ 4780us) server_negotiation.cc:407] Connection header received
0502 14:06:14.378147 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.378180 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.378504 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.378663 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:14.380466 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.382268 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.384683 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.385521 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.385704 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.386195 (+ 491us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.386422 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":1011,"threads_started":1}
I20240502 14:06:14.387861 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.369086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41285 (local address 127.0.0.1:54056)
0502 14:06:14.377229 (+ 8143us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.377484 (+ 255us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.377806 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.379292 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.379321 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.379412 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:14.380200 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.380254 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.382592 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.382633 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.384414 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.384457 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.385821 (+ 1364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.385878 (+ 57us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.385992 (+ 114us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.386249 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7510,"thread_start_us":9582,"threads_started":1}
I20240502 14:06:14.387300 2187 tablet_service.cc:1467] Processing CreateTablet for tablet 2480800d689341ee92d40be492ce980e (DEFAULT_TABLE table=test-table [id=f47450586c4a40fb9662c7ceac7d5756]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.389119 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2480800d689341ee92d40be492ce980e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.392033 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.374884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44793 (local address 127.0.0.1:45674)
0502 14:06:14.379589 (+ 4705us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.379648 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.379986 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.381323 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.381347 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.381431 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:14.382368 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.382418 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.385081 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.385112 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.386902 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.386957 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.388894 (+ 1937us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.388975 (+ 81us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.390144 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.390371 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4085,"thread_start_us":1430,"threads_started":1}
I20240502 14:06:14.391719 2329 tablet_service.cc:1467] Processing CreateTablet for tablet 2480800d689341ee92d40be492ce980e (DEFAULT_TABLE table=test-table [id=f47450586c4a40fb9662c7ceac7d5756]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.392707 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.372345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.27.174.67:44793)
0502 14:06:14.375243 (+ 2898us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.375273 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.379765 (+ 4492us) server_negotiation.cc:407] Connection header received
0502 14:06:14.380288 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.380319 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.380651 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.380813 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:14.382698 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.384859 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.387214 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.388456 (+ 1242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.388660 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.389629 (+ 969us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.389844 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":1823,"threads_started":1}
I20240502 14:06:14.393254 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2480800d689341ee92d40be492ce980e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.396675 2471 tablet_service.cc:1467] Processing CreateTablet for tablet 2480800d689341ee92d40be492ce980e (DEFAULT_TABLE table=test-table [id=f47450586c4a40fb9662c7ceac7d5756]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.398417 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2480800d689341ee92d40be492ce980e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.415339 2560 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Bootstrap starting.
I20240502 14:06:14.420469 2561 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Bootstrap starting.
I20240502 14:06:14.420895 2560 tablet_bootstrap.cc:654] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.423780 2560 log.cc:824] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:14.427031 2561 tablet_bootstrap.cc:654] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.427271 2562 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Bootstrap starting.
I20240502 14:06:14.428876 2561 log.cc:824] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:14.430505 2560 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: No bootstrap required, opened a new log
I20240502 14:06:14.431205 2560 ts_tablet_manager.cc:1367] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240502 14:06:14.433184 2562 tablet_bootstrap.cc:654] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.435807 2562 log.cc:824] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:14.440168 2561 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: No bootstrap required, opened a new log
I20240502 14:06:14.440752 2561 ts_tablet_manager.cc:1367] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240502 14:06:14.447947 2562 tablet_bootstrap.cc:492] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: No bootstrap required, opened a new log
I20240502 14:06:14.448550 2562 ts_tablet_manager.cc:1367] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20240502 14:06:14.471228 2561 raft_consensus.cc:348] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.472352 2561 raft_consensus.cc:374] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.472713 2561 raft_consensus.cc:729] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.472592 2560 raft_consensus.cc:348] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.473461 2560 raft_consensus.cc:374] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.473803 2560 raft_consensus.cc:729] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.473991 2561 consensus_queue.cc:260] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.475147 2560 consensus_queue.cc:260] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.480121 2561 ts_tablet_manager.cc:1398] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Time spent starting tablet: real 0.039s user 0.029s sys 0.007s
I20240502 14:06:14.483155 2560 ts_tablet_manager.cc:1398] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Time spent starting tablet: real 0.052s user 0.033s sys 0.005s
I20240502 14:06:14.482831 2562 raft_consensus.cc:348] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.483743 2562 raft_consensus.cc:374] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.484033 2562 raft_consensus.cc:729] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.484683 2562 consensus_queue.cc:260] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.487514 2536 heartbeater.cc:503] Master 127.27.174.126:41651 was elected leader, sending a full tablet report...
I20240502 14:06:14.490460 2562 ts_tablet_manager.cc:1398] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Time spent starting tablet: real 0.042s user 0.035s sys 0.000s
I20240502 14:06:14.579910 2568 raft_consensus.cc:482] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:14.580386 2568 raft_consensus.cc:504] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.582770 2568 leader_election.cc:288] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:14.590086 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.583183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.27.174.67:46279)
0502 14:06:14.583806 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.583847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.584071 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.585031 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.585055 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.585145 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:14.585600 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.585651 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.587197 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.587222 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.588620 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.588693 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.588984 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.589043 (+ 59us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.589128 (+ 85us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.589264 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20240502 14:06:14.591732 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.583245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:46279 (local address 127.27.174.65:32809)
0502 14:06:14.584084 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.584107 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.584154 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:06:14.584305 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.584342 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.584571 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.584733 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:14.585857 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.586911 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.588898 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.589967 (+ 1069us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.590127 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.590295 (+ 168us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.590498 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20240502 14:06:14.592903 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.583920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:57371 (local address 127.27.174.66:41285)
0502 14:06:14.584732 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.584755 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.585423 (+ 668us) server_negotiation.cc:407] Connection header received
0502 14:06:14.585761 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.585792 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.586020 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.586213 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:14.587646 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.588726 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.590874 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.591565 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.591674 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.591803 (+ 129us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.591936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20240502 14:06:14.592978 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.583428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41285 (local address 127.27.174.67:57371)
0502 14:06:14.585252 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.585302 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.585532 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.586263 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.586291 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.586385 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:14.587422 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.587456 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.588956 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.588983 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.590651 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.590692 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.590993 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.591080 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.591192 (+ 112us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.591382 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1068,"threads_started":1}
I20240502 14:06:14.593299 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2480800d689341ee92d40be492ce980e" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39" is_pre_election: true
I20240502 14:06:14.594133 2207 raft_consensus.cc:2454] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 0.
I20240502 14:06:14.594563 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2480800d689341ee92d40be492ce980e" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0" is_pre_election: true
I20240502 14:06:14.595299 2349 raft_consensus.cc:2454] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 0.
I20240502 14:06:14.595455 2417 leader_election.cc:302] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da730156fda48e7a89bbe894e2c31f9, 97b50f777bb6437aafe6fb971346ed39; no voters:
I20240502 14:06:14.596161 2568 raft_consensus.cc:2781] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:14.596565 2568 raft_consensus.cc:482] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:14.596802 2568 raft_consensus.cc:3037] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.601183 2568 raft_consensus.cc:504] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.602501 2568 leader_election.cc:288] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Requested vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:14.603168 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2480800d689341ee92d40be492ce980e" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39"
I20240502 14:06:14.603382 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2480800d689341ee92d40be492ce980e" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0"
I20240502 14:06:14.603705 2207 raft_consensus.cc:3037] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.603879 2349 raft_consensus.cc:3037] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.610738 2349 raft_consensus.cc:2454] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 1.
I20240502 14:06:14.610738 2207 raft_consensus.cc:2454] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 1.
I20240502 14:06:14.611758 2418 leader_election.cc:302] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 8da730156fda48e7a89bbe894e2c31f9; no voters:
I20240502 14:06:14.612327 2568 raft_consensus.cc:2781] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:14.613720 2568 raft_consensus.cc:686] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 1 LEADER]: Becoming Leader. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Running, Role: LEADER
I20240502 14:06:14.614519 2568 consensus_queue.cc:237] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.625078 2041 catalog_manager.cc:5484] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "8da730156fda48e7a89bbe894e2c31f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:14.651286 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.644623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.27.174.67:44793)
0502 14:06:14.645098 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.645116 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.645256 (+ 140us) server_negotiation.cc:407] Connection header received
0502 14:06:14.645588 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.645608 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.645837 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.645980 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:14.646825 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.647639 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.648787 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.649103 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.649240 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:14.649733 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:14.649881 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.650094 (+ 213us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.650256 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20240502 14:06:14.661684 2349 raft_consensus.cc:1261] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Refusing update from remote peer 8da730156fda48e7a89bbe894e2c31f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:14.661774 2207 raft_consensus.cc:1261] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Refusing update from remote peer 8da730156fda48e7a89bbe894e2c31f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:14.663210 2570 consensus_queue.cc:1035] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:14.663920 2568 consensus_queue.cc:1035] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:14.691181 2395 tablet.cc:2369] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:06:14.691489 2573 mvcc.cc:205] Tried to move back new op lower bound from 7023242341000306688 to 7023242340827963392. Current Snapshot: MvccSnapshot[applied={T|T < 7023242341000306688 or (T in {7023242341000306688})}]
I20240502 14:06:14.706243 2041 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:14.706899 2041 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:14.720407 2041 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Sending DeleteTablet for 3 replicas of tablet 2480800d689341ee92d40be492ce980e
I20240502 14:06:14.724423 2329 tablet_service.cc:1514] Processing DeleteTablet for tablet 2480800d689341ee92d40be492ce980e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:14 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54056
I20240502 14:06:14.725124 2187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2480800d689341ee92d40be492ce980e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:14 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59332
I20240502 14:06:14.728113 2583 tablet_replica.cc:330] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: stopping tablet replica
I20240502 14:06:14.728220 2471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2480800d689341ee92d40be492ce980e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:14 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:45674
I20240502 14:06:14.728860 2583 raft_consensus.cc:2229] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.729513 2582 tablet_replica.cc:330] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: stopping tablet replica
I20240502 14:06:14.729456 2583 raft_consensus.cc:2258] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.730211 2582 raft_consensus.cc:2229] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.730788 2582 raft_consensus.cc:2258] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.732901 2583 ts_tablet_manager.cc:1875] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:14.733027 2043 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:06:14.739833 2584 tablet_replica.cc:330] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: stopping tablet replica
W20240502 14:06:14.739748 2043 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:14.740621 2584 raft_consensus.cc:2229] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:14.741555 2584 raft_consensus.cc:2258] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.744904 2584 ts_tablet_manager.cc:1875] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:14.759930 2583 ts_tablet_manager.cc:1888] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:14.769670 2584 ts_tablet_manager.cc:1888] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:14.770210 2583 log.cc:1196] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/2480800d689341ee92d40be492ce980e
I20240502 14:06:14.771067 2583 ts_tablet_manager.cc:1909] T 2480800d689341ee92d40be492ce980e P 97b50f777bb6437aafe6fb971346ed39: Deleting consensus metadata
I20240502 14:06:14.771839 2187 tablet_service.cc:1467] Processing CreateTablet for tablet b81688673ca2436f9890ed72117276ec (DEFAULT_TABLE table=test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.772832 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81688673ca2436f9890ed72117276ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.774261 2028 catalog_manager.cc:4830] TS 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809): tablet 2480800d689341ee92d40be492ce980e (table test-table [id=f47450586c4a40fb9662c7ceac7d5756]) successfully deleted
I20240502 14:06:14.775419 2584 log.cc:1196] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/2480800d689341ee92d40be492ce980e
I20240502 14:06:14.776240 2584 ts_tablet_manager.cc:1909] T 2480800d689341ee92d40be492ce980e P 8da730156fda48e7a89bbe894e2c31f9: Deleting consensus metadata
I20240502 14:06:14.779024 2028 catalog_manager.cc:4830] TS 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793): tablet 2480800d689341ee92d40be492ce980e (table test-table [id=f47450586c4a40fb9662c7ceac7d5756]) successfully deleted
I20240502 14:06:14.780284 2329 tablet_service.cc:1467] Processing CreateTablet for tablet b81688673ca2436f9890ed72117276ec (DEFAULT_TABLE table=test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.781325 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81688673ca2436f9890ed72117276ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.784029 2582 ts_tablet_manager.cc:1875] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:14.788679 2471 tablet_service.cc:1467] Processing CreateTablet for tablet b81688673ca2436f9890ed72117276ec (DEFAULT_TABLE table=test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:14.789777 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81688673ca2436f9890ed72117276ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.797382 2582 ts_tablet_manager.cc:1888] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:14.797843 2582 log.cc:1196] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/2480800d689341ee92d40be492ce980e
I20240502 14:06:14.798193 2560 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Bootstrap starting.
I20240502 14:06:14.798877 2582 ts_tablet_manager.cc:1909] T 2480800d689341ee92d40be492ce980e P 71e426200d0549cc8cab3b54553d69d0: Deleting consensus metadata
I20240502 14:06:14.802500 2029 catalog_manager.cc:4830] TS 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): tablet 2480800d689341ee92d40be492ce980e (table test-table [id=f47450586c4a40fb9662c7ceac7d5756]) successfully deleted
I20240502 14:06:14.806574 2560 tablet_bootstrap.cc:654] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.811129 2561 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Bootstrap starting.
I20240502 14:06:14.819008 2562 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Bootstrap starting.
I20240502 14:06:14.821018 2561 tablet_bootstrap.cc:654] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.824388 2562 tablet_bootstrap.cc:654] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.836812 2562 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: No bootstrap required, opened a new log
I20240502 14:06:14.837339 2562 ts_tablet_manager.cc:1367] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.011s
I20240502 14:06:14.839970 2562 raft_consensus.cc:348] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.840768 2562 raft_consensus.cc:374] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.841039 2562 raft_consensus.cc:729] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.841639 2562 consensus_queue.cc:260] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.843710 2562 ts_tablet_manager.cc:1398] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:14.845377 2561 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: No bootstrap required, opened a new log
I20240502 14:06:14.845490 2570 raft_consensus.cc:482] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:14.845816 2561 ts_tablet_manager.cc:1367] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.009s
I20240502 14:06:14.845883 2570 raft_consensus.cc:504] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.847954 2570 leader_election.cc:288] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:14.848114 2561 raft_consensus.cc:348] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.848714 2561 raft_consensus.cc:374] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.848933 2561 raft_consensus.cc:729] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.849637 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81688673ca2436f9890ed72117276ec" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0" is_pre_election: true
I20240502 14:06:14.849715 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81688673ca2436f9890ed72117276ec" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39" is_pre_election: true
W20240502 14:06:14.851310 2417 leader_election.cc:341] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:14.852583 2560 tablet_bootstrap.cc:492] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: No bootstrap required, opened a new log
I20240502 14:06:14.852973 2560 ts_tablet_manager.cc:1367] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20240502 14:06:14.855095 2560 raft_consensus.cc:348] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.855741 2560 raft_consensus.cc:374] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.855980 2560 raft_consensus.cc:729] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.856560 2560 consensus_queue.cc:260] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.858624 2560 ts_tablet_manager.cc:1398] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:06:14.849524 2561 consensus_queue.cc:260] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.859479 2349 raft_consensus.cc:2454] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 0.
I20240502 14:06:14.860623 2561 ts_tablet_manager.cc:1398] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240502 14:06:14.862766 2418 leader_election.cc:302] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 8da730156fda48e7a89bbe894e2c31f9; no voters: 97b50f777bb6437aafe6fb971346ed39
I20240502 14:06:14.866483 2570 raft_consensus.cc:2781] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:14.866809 2570 raft_consensus.cc:482] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:14.867111 2570 raft_consensus.cc:3037] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.874128 2570 raft_consensus.cc:504] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.876441 2570 leader_election.cc:288] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Requested vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:14.876518 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81688673ca2436f9890ed72117276ec" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39"
I20240502 14:06:14.876674 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81688673ca2436f9890ed72117276ec" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0"
I20240502 14:06:14.877096 2207 raft_consensus.cc:3037] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.877187 2349 raft_consensus.cc:3037] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.880976 2349 raft_consensus.cc:2454] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 1.
I20240502 14:06:14.881688 2418 leader_election.cc:302] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 8da730156fda48e7a89bbe894e2c31f9; no voters:
I20240502 14:06:14.882272 2570 raft_consensus.cc:2781] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:14.882716 2570 raft_consensus.cc:686] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 1 LEADER]: Becoming Leader. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Running, Role: LEADER
I20240502 14:06:14.883174 2207 raft_consensus.cc:2454] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 1.
I20240502 14:06:14.883425 2570 consensus_queue.cc:237] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:14.890491 2043 catalog_manager.cc:5484] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "8da730156fda48e7a89bbe894e2c31f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } health_report { overall_health: HEALTHY } } }
May 02 14:06:14 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658768, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.126@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:14.951983 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.928859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.27.174.126:41651)
0502 14:06:14.931406 (+ 2547us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:14.931421 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:14.931752 (+ 331us) server_negotiation.cc:407] Connection header received
0502 14:06:14.932153 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:14.932169 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:14.932318 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:14.932441 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:14.934530 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.935442 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.937160 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.937527 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.942309 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:14.942398 (+ 89us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:14.942422 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:14.942550 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:14.946143 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.947495 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.947514 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.947527 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.947649 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:14.948180 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:14.948197 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:14.948215 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:14.948559 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:14.948743 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:14.949510 (+ 767us) server_negotiation.cc:299] Negotiation successful
0502 14:06:14.950072 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":207,"server-negotiator.queue_time_us":2028,"thread_start_us":1723,"threads_started":1}
I20240502 14:06:14.952260 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:14.928745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41651 (local address 127.0.0.1:47422)
0502 14:06:14.931591 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:14.931637 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:14.931924 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:14.932557 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:14.932584 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:14.933683 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:14.934288 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.934334 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.935630 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:14.935660 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:14.936856 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:14.936908 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:14.937231 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:14.938237 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:14.938331 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:14.942057 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:14.946371 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.946408 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.946434 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.947307 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.947786 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:14.947813 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:14.947837 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:14.948040 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:14.948752 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:14.948784 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:14.949079 (+ 295us) client_negotiation.cc:769] Sending connection context
0502 14:06:14.949357 (+ 278us) client_negotiation.cc:240] Negotiation successful
0502 14:06:14.949830 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2245,"thread_start_us":1905,"threads_started":1}
W20240502 14:06:14.956362 2253 tablet.cc:2369] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:06:14.958971 2041 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47422:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240502 14:06:14.960706 2041 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:14.977507 2187 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c3791eda4807b6915ec3c5c9dd01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1128ba4b18da46b8b4997fb73d17f0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:06:14.978683 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c3791eda4807b6915ec3c5c9dd01. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:06:14.978686 2537 tablet.cc:2369] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:06:14.979374 2471 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c3791eda4807b6915ec3c5c9dd01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1128ba4b18da46b8b4997fb73d17f0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:06:14.980262 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c3791eda4807b6915ec3c5c9dd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.983119 2329 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c3791eda4807b6915ec3c5c9dd01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1128ba4b18da46b8b4997fb73d17f0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240502 14:06:14.984239 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c3791eda4807b6915ec3c5c9dd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.992678 2560 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: Bootstrap starting.
I20240502 14:06:14.997849 2560 tablet_bootstrap.cc:654] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.997884 2562 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: Bootstrap starting.
I20240502 14:06:14.998867 2561 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: Bootstrap starting.
I20240502 14:06:15.003073 2562 tablet_bootstrap.cc:654] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.004992 2561 tablet_bootstrap.cc:654] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.013569 2560 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: No bootstrap required, opened a new log
I20240502 14:06:15.014087 2560 ts_tablet_manager.cc:1367] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20240502 14:06:15.016999 2560 raft_consensus.cc:348] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.017943 2560 raft_consensus.cc:374] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.019198 2560 raft_consensus.cc:729] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.020304 2561 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: No bootstrap required, opened a new log
I20240502 14:06:15.019869 2560 consensus_queue.cc:260] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.020726 2561 ts_tablet_manager.cc:1367] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.008s
I20240502 14:06:15.021184 2567 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.022020 2567 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.022045 2562 tablet_bootstrap.cc:492] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: No bootstrap required, opened a new log
I20240502 14:06:15.022586 2562 ts_tablet_manager.cc:1367] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.008s
I20240502 14:06:15.023130 2561 raft_consensus.cc:348] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.023814 2561 raft_consensus.cc:374] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.024168 2561 raft_consensus.cc:729] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.024940 2561 consensus_queue.cc:260] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.026300 2566 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.027067 2566 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.024811 2562 raft_consensus.cc:348] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.027714 2562 raft_consensus.cc:374] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.027971 2562 raft_consensus.cc:729] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.028146 2560 ts_tablet_manager.cc:1398] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:06:15.028609 2562 consensus_queue.cc:260] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.032639 2561 ts_tablet_manager.cc:1398] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:06:15.029659 2570 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.034255 2570 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.035779 2562 ts_tablet_manager.cc:1398] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240502 14:06:15.051185 2567 raft_consensus.cc:482] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:15.051539 2567 raft_consensus.cc:504] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.053560 2567 leader_election.cc:288] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:15.066459 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.054996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57955 (local address 127.27.174.66:41285)
0502 14:06:15.055623 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.055647 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.057312 (+ 1665us) server_negotiation.cc:407] Connection header received
0502 14:06:15.057704 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.057729 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.057954 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.058127 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:15.059376 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.060853 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.063236 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.064304 (+ 1068us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.064437 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.065052 (+ 615us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.065252 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20240502 14:06:15.068791 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.053969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41285 (local address 127.27.174.65:57955)
0502 14:06:15.057103 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.057159 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.057442 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.058170 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.058195 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.058292 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:15.059152 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.059187 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.061184 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.061212 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.062903 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.062952 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.064667 (+ 1715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.064763 (+ 96us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.066955 (+ 2192us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.067154 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":2154,"threads_started":1}
I20240502 14:06:15.069483 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.054060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44793 (local address 127.27.174.65:39891)
0502 14:06:15.057627 (+ 3567us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.057660 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.057850 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.058556 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.058583 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.058675 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:15.059899 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.059947 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.063774 (+ 3827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.063803 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.065434 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.065483 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.067380 (+ 1897us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.067497 (+ 117us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.067634 (+ 137us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.067818 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2935,"thread_start_us":2600,"threads_started":1}
I20240502 14:06:15.070560 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.054409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39891 (local address 127.27.174.67:44793)
0502 14:06:15.055178 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.055208 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.057752 (+ 2544us) server_negotiation.cc:407] Connection header received
0502 14:06:15.058032 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.058091 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.058335 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.058492 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:15.060166 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.063416 (+ 3250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.065706 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.066829 (+ 1123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.066972 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.067765 (+ 793us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.067997 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20240502 14:06:15.070341 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0" is_pre_election: true
I20240502 14:06:15.071040 2349 raft_consensus.cc:2454] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97b50f777bb6437aafe6fb971346ed39 in term 0.
I20240502 14:06:15.072311 2570 raft_consensus.cc:482] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:15.072496 2134 leader_election.cc:302] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 97b50f777bb6437aafe6fb971346ed39; no voters:
I20240502 14:06:15.072729 2570 raft_consensus.cc:504] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.073235 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9" is_pre_election: true
I20240502 14:06:15.073971 2491 raft_consensus.cc:2454] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97b50f777bb6437aafe6fb971346ed39 in term 0.
I20240502 14:06:15.074594 2570 leader_election.cc:288] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:15.074785 2567 raft_consensus.cc:2781] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:15.075102 2567 raft_consensus.cc:482] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:15.075415 2567 raft_consensus.cc:3037] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.076185 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0" is_pre_election: true
I20240502 14:06:15.076628 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39" is_pre_election: true
I20240502 14:06:15.076781 2349 raft_consensus.cc:2454] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da730156fda48e7a89bbe894e2c31f9 in term 0.
I20240502 14:06:15.077843 2418 leader_election.cc:302] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 8da730156fda48e7a89bbe894e2c31f9; no voters:
I20240502 14:06:15.078629 2570 raft_consensus.cc:2781] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:15.078936 2570 raft_consensus.cc:482] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:15.079221 2570 raft_consensus.cc:3037] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.081804 2567 raft_consensus.cc:504] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.082943 2207 raft_consensus.cc:2379] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8da730156fda48e7a89bbe894e2c31f9 in current term 1: Already voted for candidate 97b50f777bb6437aafe6fb971346ed39 in this term.
I20240502 14:06:15.083590 2567 leader_election.cc:288] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 election: Requested vote from peers 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:15.085268 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9"
I20240502 14:06:15.086197 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0"
I20240502 14:06:15.086787 2349 raft_consensus.cc:3037] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.088543 2570 raft_consensus.cc:504] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.089705 2491 raft_consensus.cc:2379] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97b50f777bb6437aafe6fb971346ed39 in current term 1: Already voted for candidate 8da730156fda48e7a89bbe894e2c31f9 in this term.
I20240502 14:06:15.090422 2570 leader_election.cc:288] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Requested vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285)
I20240502 14:06:15.091490 2348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0"
I20240502 14:06:15.093636 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c3791eda4807b6915ec3c5c9dd01" candidate_uuid: "8da730156fda48e7a89bbe894e2c31f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39"
I20240502 14:06:15.094422 2207 raft_consensus.cc:2379] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8da730156fda48e7a89bbe894e2c31f9 in current term 1: Already voted for candidate 97b50f777bb6437aafe6fb971346ed39 in this term.
I20240502 14:06:15.094659 2349 raft_consensus.cc:2454] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97b50f777bb6437aafe6fb971346ed39 in term 1.
I20240502 14:06:15.095425 2417 leader_election.cc:302] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8da730156fda48e7a89bbe894e2c31f9; no voters: 71e426200d0549cc8cab3b54553d69d0, 97b50f777bb6437aafe6fb971346ed39
I20240502 14:06:15.095743 2134 leader_election.cc:302] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 97b50f777bb6437aafe6fb971346ed39; no voters: 8da730156fda48e7a89bbe894e2c31f9
I20240502 14:06:15.096153 2570 raft_consensus.cc:2726] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:06:15.096628 2567 raft_consensus.cc:2781] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:15.098053 2567 raft_consensus.cc:686] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [term 1 LEADER]: Becoming Leader. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Running, Role: LEADER
I20240502 14:06:15.098935 2567 consensus_queue.cc:237] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.105154 2601 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: TxnStatusTablet state changed. Reason: New leader 97b50f777bb6437aafe6fb971346ed39. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.105990 2601 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:06:15.108278 2603 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240502 14:06:15.108661 2603 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240502 14:06:15.109769 2043 catalog_manager.cc:5484] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:15.156395 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.142930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.0.0.1:59336)
0502 14:06:15.143950 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.143991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.144281 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.148194 (+ 3913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.148225 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.148275 (+ 50us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:06:15.148754 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.148800 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.150870 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.150902 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.152123 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.152179 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.152483 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.152610 (+ 127us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.154527 (+ 1917us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.154571 (+ 44us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.154715 (+ 144us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.154922 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383}
I20240502 14:06:15.156561 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.143157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.27.174.65:32809)
0502 14:06:15.147345 (+ 4188us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.147369 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.147417 (+ 48us) server_negotiation.cc:407] Connection header received
0502 14:06:15.147618 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.147642 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.147927 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.148104 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:15.149027 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.150629 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.152941 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.153501 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.153665 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.154355 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.154523 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.154766 (+ 243us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.154905 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3561,"thread_start_us":3238,"threads_started":1}
I20240502 14:06:15.164829 2491 raft_consensus.cc:1261] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Refusing update from remote peer 97b50f777bb6437aafe6fb971346ed39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.165030 2349 raft_consensus.cc:1261] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Refusing update from remote peer 97b50f777bb6437aafe6fb971346ed39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.166390 2567 consensus_queue.cc:1035] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:15.167248 2601 consensus_queue.cc:1035] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:15.189149 2566 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: TxnStatusTablet state changed. Reason: New leader 97b50f777bb6437aafe6fb971346ed39. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.190029 2566 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.194626 2567 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.195286 2567 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:06:15.195828 2574 mvcc.cc:205] Tried to move back new op lower bound from 7023242343060070400 to 7023242342815166464. Current Snapshot: MvccSnapshot[applied={T|T < 7023242343060070400}]
I20240502 14:06:15.198197 2575 mvcc.cc:205] Tried to move back new op lower bound from 7023242343060070400 to 7023242342815166464. Current Snapshot: MvccSnapshot[applied={T|T < 7023242343060070400 or (T in {7023242343060070400})}]
I20240502 14:06:15.199918 2566 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.211203 2566 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 71e426200d0549cc8cab3b54553d69d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.234344 2570 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: TxnStatusTablet state changed. Reason: New leader 97b50f777bb6437aafe6fb971346ed39. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.235055 2570 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.241294 2567 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.241995 2567 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 97b50f777bb6437aafe6fb971346ed39: This TxnStatusTablet replica's current role is: LEADER
I20240502 14:06:15.249895 2570 tablet_replica.cc:439] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } } }
I20240502 14:06:15.252552 2570 tablet_replica.cc:442] T a3e6c3791eda4807b6915ec3c5c9dd01 P 8da730156fda48e7a89bbe894e2c31f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240502 14:06:15.286517 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.259452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.27.174.67:50161)
0502 14:06:15.275768 (+ 16316us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.275832 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.276152 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.278333 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.278360 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.278416 (+ 56us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:06:15.279212 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.279273 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.281000 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.281032 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.282551 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.282613 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.282935 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.283064 (+ 129us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.284562 (+ 1498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.284600 (+ 38us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.284714 (+ 114us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.284889 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15588,"thread_start_us":14904,"threads_started":1}
I20240502 14:06:15.288621 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.261084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50161 (local address 127.27.174.65:32809)
0502 14:06:15.261761 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.261788 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.275981 (+ 14193us) server_negotiation.cc:407] Connection header received
0502 14:06:15.276429 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.276461 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.276761 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.277931 (+ 1170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:15.279564 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.280754 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.283332 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.283872 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.284028 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.284311 (+ 283us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.284457 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.287042 (+ 2585us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.287222 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20240502 14:06:15.327874 2349 raft_consensus.cc:1261] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Refusing update from remote peer 8da730156fda48e7a89bbe894e2c31f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.329211 2207 raft_consensus.cc:1261] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Refusing update from remote peer 8da730156fda48e7a89bbe894e2c31f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:15.329213 2570 consensus_queue.cc:1035] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:15.330598 2570 consensus_queue.cc:1035] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:15.449134 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.428473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:43015 (local address 127.27.174.67:44793)
0502 14:06:15.429251 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.429284 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.436328 (+ 7044us) server_negotiation.cc:407] Connection header received
0502 14:06:15.438363 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.438396 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.438702 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.438875 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:15.442238 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.443536 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.445626 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.446141 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.446837 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.447105 (+ 268us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.447230 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.447634 (+ 404us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.447787 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20240502 14:06:15.451907 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.428074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44793 (local address 127.27.174.65:43015)
0502 14:06:15.436134 (+ 8060us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.436184 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.438088 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.439016 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.439045 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.439102 (+ 57us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:06:15.440224 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.440272 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.443886 (+ 3614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.443920 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.445347 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.445400 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.446456 (+ 1056us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.446578 (+ 122us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:06:15.447383 (+ 805us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:06:15.447419 (+ 36us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.449731 (+ 2312us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.449947 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7248,"thread_start_us":6913,"threads_started":1}
I20240502 14:06:15.571763 2043 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:15.572266 2043 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47412:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:15.581504 2043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Sending DeleteTablet for 3 replicas of tablet b81688673ca2436f9890ed72117276ec
I20240502 14:06:15.583509 2329 tablet_service.cc:1514] Processing DeleteTablet for tablet b81688673ca2436f9890ed72117276ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:15 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54056
I20240502 14:06:15.587822 2471 tablet_service.cc:1514] Processing DeleteTablet for tablet b81688673ca2436f9890ed72117276ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:15 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:45674
I20240502 14:06:15.600054 2187 tablet_service.cc:1514] Processing DeleteTablet for tablet b81688673ca2436f9890ed72117276ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:15 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59332
I20240502 14:06:15.606973 2621 tablet_replica.cc:330] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: stopping tablet replica
I20240502 14:06:15.608036 2621 raft_consensus.cc:2229] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:15.609239 2621 raft_consensus.cc:2258] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:15.611089 2622 tablet_replica.cc:330] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: stopping tablet replica
I20240502 14:06:15.612026 2622 raft_consensus.cc:2229] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:15.612080 2620 tablet_replica.cc:330] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: stopping tablet replica
I20240502 14:06:15.612625 2622 raft_consensus.cc:2258] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:15.614287 2620 raft_consensus.cc:2229] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:15.614833 2620 raft_consensus.cc:2258] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:15.615926 2621 ts_tablet_manager.cc:1875] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:15.616340 2622 ts_tablet_manager.cc:1875] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:15.624248 2620 ts_tablet_manager.cc:1875] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:15.645686 2620 ts_tablet_manager.cc:1888] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:06:15.646214 2620 log.cc:1196] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/b81688673ca2436f9890ed72117276ec
I20240502 14:06:15.647051 2620 ts_tablet_manager.cc:1909] T b81688673ca2436f9890ed72117276ec P 71e426200d0549cc8cab3b54553d69d0: Deleting consensus metadata
I20240502 14:06:15.648777 2622 ts_tablet_manager.cc:1888] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:06:15.649219 2622 log.cc:1196] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/b81688673ca2436f9890ed72117276ec
I20240502 14:06:15.649719 2029 catalog_manager.cc:4830] TS 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): tablet b81688673ca2436f9890ed72117276ec (table test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]) successfully deleted
I20240502 14:06:15.650060 2622 ts_tablet_manager.cc:1909] T b81688673ca2436f9890ed72117276ec P 97b50f777bb6437aafe6fb971346ed39: Deleting consensus metadata
I20240502 14:06:15.652631 2028 catalog_manager.cc:4830] TS 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809): tablet b81688673ca2436f9890ed72117276ec (table test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]) successfully deleted
I20240502 14:06:15.654651 2621 ts_tablet_manager.cc:1888] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:06:15.655067 2621 log.cc:1196] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/b81688673ca2436f9890ed72117276ec
I20240502 14:06:15.655858 2621 ts_tablet_manager.cc:1909] T b81688673ca2436f9890ed72117276ec P 8da730156fda48e7a89bbe894e2c31f9: Deleting consensus metadata
I20240502 14:06:15.658674 2028 catalog_manager.cc:4830] TS 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793): tablet b81688673ca2436f9890ed72117276ec (table test-table [id=cc1c1626243e4b5aa800fa8c318f8c48]) successfully deleted
May 02 14:06:15 dist-test-slave-dlb8 krb5kdc[1995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 02 14:06:15 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:15 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:15.705070 2629 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:15.681879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.27.174.126:41651)
0502 14:06:15.688680 (+ 6801us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.688694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.688730 (+ 36us) server_negotiation.cc:407] Connection header received
0502 14:06:15.688834 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.688850 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.689020 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.689184 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:15.691730 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.692602 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.695667 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.696197 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.703641 (+ 7444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47424: BlockingRecv error: recv got EOF from 127.0.0.1:47424 (error 108)
Metrics: {"server-negotiator.queue_time_us":6378,"thread_start_us":6108,"threads_started":1}
May 02 14:06:15 dist-test-slave-dlb8 krb5kdc[1995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM
I20240502 14:06:15.780318 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.741335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.27.174.126:41651)
0502 14:06:15.742093 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.742121 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.750765 (+ 8644us) server_negotiation.cc:407] Connection header received
0502 14:06:15.751594 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.751622 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.751839 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.751978 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:15.757568 (+ 5590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.758807 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.760863 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.761396 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.767948 (+ 6552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:15.768048 (+ 100us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:15.768076 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:15.768214 (+ 138us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:15.773220 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:15.774969 (+ 1749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:15.774999 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:15.775020 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:15.775198 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:15.775910 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:15.775937 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:15.775963 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:15.776497 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:15.776846 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.777837 (+ 991us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.778302 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20240502 14:06:15.787766 2043 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:15.789372 2043 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:15.815207 2187 tablet_service.cc:1467] Processing CreateTablet for tablet 5305d46c75af4eab969a0e15778abc21 (DEFAULT_TABLE table=test-table [id=c2b9dc26492c4cd6af981d15ce632983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.816355 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5305d46c75af4eab969a0e15778abc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.825098 2329 tablet_service.cc:1467] Processing CreateTablet for tablet 5305d46c75af4eab969a0e15778abc21 (DEFAULT_TABLE table=test-table [id=c2b9dc26492c4cd6af981d15ce632983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.826337 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5305d46c75af4eab969a0e15778abc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.840065 2471 tablet_service.cc:1467] Processing CreateTablet for tablet 5305d46c75af4eab969a0e15778abc21 (DEFAULT_TABLE table=test-table [id=c2b9dc26492c4cd6af981d15ce632983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.841127 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5305d46c75af4eab969a0e15778abc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.882289 2637 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Bootstrap starting.
I20240502 14:06:15.883656 2638 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Bootstrap starting.
I20240502 14:06:15.883831 2636 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Bootstrap starting.
I20240502 14:06:15.888969 2638 tablet_bootstrap.cc:654] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.899928 2638 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: No bootstrap required, opened a new log
I20240502 14:06:15.900404 2636 tablet_bootstrap.cc:654] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.900460 2638 ts_tablet_manager.cc:1367] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240502 14:06:15.901908 2637 tablet_bootstrap.cc:654] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.903141 2638 raft_consensus.cc:348] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.903942 2638 raft_consensus.cc:374] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.904235 2638 raft_consensus.cc:729] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.904902 2638 consensus_queue.cc:260] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.912374 2638 ts_tablet_manager.cc:1398] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Time spent starting tablet: real 0.011s user 0.000s sys 0.004s
I20240502 14:06:15.919095 2637 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: No bootstrap required, opened a new log
I20240502 14:06:15.919561 2637 ts_tablet_manager.cc:1367] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.006s
I20240502 14:06:15.919625 2636 tablet_bootstrap.cc:492] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: No bootstrap required, opened a new log
I20240502 14:06:15.920090 2636 ts_tablet_manager.cc:1367] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.007s
I20240502 14:06:15.921897 2637 raft_consensus.cc:348] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.922637 2637 raft_consensus.cc:374] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.922480 2636 raft_consensus.cc:348] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.922927 2637 raft_consensus.cc:729] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.923331 2636 raft_consensus.cc:374] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.923624 2636 raft_consensus.cc:729] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.923583 2637 consensus_queue.cc:260] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.925693 2637 ts_tablet_manager.cc:1398] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240502 14:06:15.924333 2636 consensus_queue.cc:260] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.941787 2636 ts_tablet_manager.cc:1398] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Time spent starting tablet: real 0.021s user 0.004s sys 0.014s
I20240502 14:06:15.950134 2642 raft_consensus.cc:482] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:15.950608 2642 raft_consensus.cc:504] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:15.959156 2642 leader_election.cc:288] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:15.993266 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.953671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39885 (local address 127.27.174.65:32809)
0502 14:06:15.966427 (+ 12756us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.966456 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:15.975448 (+ 8992us) server_negotiation.cc:407] Connection header received
0502 14:06:15.975606 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:15.975632 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:15.975866 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:15.976008 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:15.982399 (+ 6391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.983674 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.990187 (+ 6513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.991348 (+ 1161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.991492 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:15.991667 (+ 175us) server_negotiation.cc:299] Negotiation successful
0502 14:06:15.991856 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12125,"thread_start_us":12740,"threads_started":1}
I20240502 14:06:15.997545 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.953288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.27.174.66:39885)
0502 14:06:15.974908 (+ 21620us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:15.974969 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:15.975250 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:15.976994 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:15.977023 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:15.977122 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:15.977930 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.982020 (+ 4090us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.985163 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:15.985202 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:15.987936 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:15.987983 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:15.988292 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:15.988374 (+ 82us) client_negotiation.cc:769] Sending connection context
0502 14:06:15.988490 (+ 116us) client_negotiation.cc:240] Negotiation successful
0502 14:06:15.988672 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13796,"thread_start_us":13525,"threads_started":1}
I20240502 14:06:16.010344 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5305d46c75af4eab969a0e15778abc21" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39" is_pre_election: true
I20240502 14:06:16.010973 2206 raft_consensus.cc:2454] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e426200d0549cc8cab3b54553d69d0 in term 0.
I20240502 14:06:16.014022 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.967921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:44793 (local address 127.27.174.66:56567)
0502 14:06:16.000212 (+ 32291us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:16.000260 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:16.000696 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:16.001831 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:16.001857 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:16.001955 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:06:16.002639 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:16.002693 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.004515 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.004550 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:16.008381 (+ 3831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:16.008434 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.008795 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.008883 (+ 88us) client_negotiation.cc:769] Sending connection context
0502 14:06:16.012309 (+ 3426us) client_negotiation.cc:240] Negotiation successful
0502 14:06:16.012513 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31629,"thread_start_us":39314,"threads_started":1}
I20240502 14:06:16.014931 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:15.967904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:56567 (local address 127.27.174.67:44793)
0502 14:06:15.999070 (+ 31166us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:15.999098 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:16.000617 (+ 1519us) server_negotiation.cc:407] Connection header received
0502 14:06:16.000994 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:16.001020 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:16.001359 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:16.001508 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:06:16.002962 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.004222 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.011892 (+ 7670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.013050 (+ 1158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.013184 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:16.013358 (+ 174us) server_negotiation.cc:299] Negotiation successful
0502 14:06:16.013560 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30564,"thread_start_us":31153,"threads_started":1}
I20240502 14:06:16.015165 2275 leader_election.cc:302] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 97b50f777bb6437aafe6fb971346ed39; no voters:
I20240502 14:06:16.016037 2642 raft_consensus.cc:2781] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:16.016358 2642 raft_consensus.cc:482] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:16.016657 2642 raft_consensus.cc:3037] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:16.016979 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5305d46c75af4eab969a0e15778abc21" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9" is_pre_election: true
I20240502 14:06:16.017596 2490 raft_consensus.cc:2454] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71e426200d0549cc8cab3b54553d69d0 in term 0.
I20240502 14:06:16.023034 2642 raft_consensus.cc:504] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.025444 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5305d46c75af4eab969a0e15778abc21" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39"
I20240502 14:06:16.025655 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5305d46c75af4eab969a0e15778abc21" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9"
I20240502 14:06:16.025949 2206 raft_consensus.cc:3037] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:16.026140 2490 raft_consensus.cc:3037] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:16.032555 2490 raft_consensus.cc:2454] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e426200d0549cc8cab3b54553d69d0 in term 1.
I20240502 14:06:16.032855 2206 raft_consensus.cc:2454] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e426200d0549cc8cab3b54553d69d0 in term 1.
I20240502 14:06:16.033686 2275 leader_election.cc:302] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0, 8da730156fda48e7a89bbe894e2c31f9; no voters:
I20240502 14:06:16.034201 2642 leader_election.cc:288] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 election: Requested vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:16.034788 2642 raft_consensus.cc:2781] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:16.035343 2642 raft_consensus.cc:686] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 1 LEADER]: Becoming Leader. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Running, Role: LEADER
I20240502 14:06:16.036311 2642 consensus_queue.cc:237] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.047194 2043 catalog_manager.cc:5484] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "71e426200d0549cc8cab3b54553d69d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:16.117218 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:16.107228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.27.174.66:41285)
0502 14:06:16.109971 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:16.109984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:16.110017 (+ 33us) server_negotiation.cc:407] Connection header received
0502 14:06:16.110178 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:16.110193 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:16.110432 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:16.110596 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:16.111448 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.112726 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.114055 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.114558 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.114708 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:16.115186 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:16.115346 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:16.115594 (+ 248us) server_negotiation.cc:299] Negotiation successful
0502 14:06:16.115749 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2363,"thread_start_us":2068,"threads_started":1}
I20240502 14:06:16.124303 2206 raft_consensus.cc:1261] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Refusing update from remote peer 71e426200d0549cc8cab3b54553d69d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:16.124303 2490 raft_consensus.cc:1261] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Refusing update from remote peer 71e426200d0549cc8cab3b54553d69d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:16.125665 2642 consensus_queue.cc:1035] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:16.126708 2647 consensus_queue.cc:1035] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:16.187565 2043 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:16.188144 2043 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:16.203586 2187 tablet_service.cc:1514] Processing DeleteTablet for tablet 5305d46c75af4eab969a0e15778abc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:16 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59332
I20240502 14:06:16.208920 2043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Sending DeleteTablet for 3 replicas of tablet 5305d46c75af4eab969a0e15778abc21
I20240502 14:06:16.210908 2186 tablet_service.cc:1514] Processing DeleteTablet for tablet 5305d46c75af4eab969a0e15778abc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:16 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59332
I20240502 14:06:16.213128 2329 tablet_service.cc:1514] Processing DeleteTablet for tablet 5305d46c75af4eab969a0e15778abc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:16 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54056
I20240502 14:06:16.223469 2471 tablet_service.cc:1514] Processing DeleteTablet for tablet 5305d46c75af4eab969a0e15778abc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:16 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:45674
I20240502 14:06:16.238577 2041 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:16.240864 2041 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:16.263749 2186 tablet_service.cc:1467] Processing CreateTablet for tablet 473e73b7ee614ec1951b2bf0570da285 (DEFAULT_TABLE table=test-table [id=c3e7f656e15347a2b0caa8e4af7ee64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:16.264853 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473e73b7ee614ec1951b2bf0570da285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:16.269201 2328 tablet_service.cc:1467] Processing CreateTablet for tablet 473e73b7ee614ec1951b2bf0570da285 (DEFAULT_TABLE table=test-table [id=c3e7f656e15347a2b0caa8e4af7ee64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:16.270422 2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473e73b7ee614ec1951b2bf0570da285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:16.284479 2470 tablet_service.cc:1467] Processing CreateTablet for tablet 473e73b7ee614ec1951b2bf0570da285 (DEFAULT_TABLE table=test-table [id=c3e7f656e15347a2b0caa8e4af7ee64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:16.285539 2470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473e73b7ee614ec1951b2bf0570da285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:16.319453 2662 tablet_replica.cc:330] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: stopping tablet replica
I20240502 14:06:16.320266 2662 raft_consensus.cc:2229] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:16.336794 2661 tablet_replica.cc:330] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: stopping tablet replica
I20240502 14:06:16.337852 2661 raft_consensus.cc:2229] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:16.340906 2660 tablet_replica.cc:330] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: stopping tablet replica
I20240502 14:06:16.341701 2660 raft_consensus.cc:2229] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:16.342331 2660 raft_consensus.cc:2258] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:16.329092 2638 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Bootstrap starting.
I20240502 14:06:16.349400 2638 tablet_bootstrap.cc:654] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:16.354132 2662 raft_consensus.cc:2258] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:16.357707 2661 raft_consensus.cc:2258] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:16.361866 2662 ts_tablet_manager.cc:1875] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:16.363869 2638 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: No bootstrap required, opened a new log
I20240502 14:06:16.364277 2638 ts_tablet_manager.cc:1367] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.007s
I20240502 14:06:16.366657 2638 raft_consensus.cc:348] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.367412 2638 raft_consensus.cc:374] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:16.367760 2638 raft_consensus.cc:729] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da730156fda48e7a89bbe894e2c31f9, State: Initialized, Role: FOLLOWER
I20240502 14:06:16.368516 2638 consensus_queue.cc:260] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.379647 2660 ts_tablet_manager.cc:1875] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:16.389356 2637 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39: Bootstrap starting.
I20240502 14:06:16.391505 2661 ts_tablet_manager.cc:1875] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:16.392869 2636 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Bootstrap starting.
I20240502 14:06:16.395226 2637 tablet_bootstrap.cc:654] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:16.396301 2662 ts_tablet_manager.cc:1888] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:16.396732 2662 log.cc:1196] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/5305d46c75af4eab969a0e15778abc21
I20240502 14:06:16.397524 2662 ts_tablet_manager.cc:1909] T 5305d46c75af4eab969a0e15778abc21 P 8da730156fda48e7a89bbe894e2c31f9: Deleting consensus metadata
I20240502 14:06:16.399991 2028 catalog_manager.cc:4830] TS 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793): tablet 5305d46c75af4eab969a0e15778abc21 (table test-table [id=c2b9dc26492c4cd6af981d15ce632983]) successfully deleted
I20240502 14:06:16.401623 2636 tablet_bootstrap.cc:654] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:16.408049 2638 ts_tablet_manager.cc:1398] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Time spent starting tablet: real 0.043s user 0.000s sys 0.005s
I20240502 14:06:16.409348 2637 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39: No bootstrap required, opened a new log
I20240502 14:06:16.409790 2637 ts_tablet_manager.cc:1367] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.001s
I20240502 14:06:16.412123 2637 raft_consensus.cc:348] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.412993 2637 raft_consensus.cc:374] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:16.413293 2637 raft_consensus.cc:729] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Initialized, Role: FOLLOWER
I20240502 14:06:16.413982 2637 consensus_queue.cc:260] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.416142 2601 raft_consensus.cc:482] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:16.416548 2601 raft_consensus.cc:504] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.418377 2601 leader_election.cc:288] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:16.419895 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9" is_pre_election: true
I20240502 14:06:16.420259 2348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0" is_pre_election: true
I20240502 14:06:16.420312 2637 ts_tablet_manager.cc:1398] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:06:16.420517 2490 raft_consensus.cc:2454] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97b50f777bb6437aafe6fb971346ed39 in term 0.
I20240502 14:06:16.421779 2133 leader_election.cc:302] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da730156fda48e7a89bbe894e2c31f9, 97b50f777bb6437aafe6fb971346ed39; no voters:
W20240502 14:06:16.422582 2134 leader_election.cc:341] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:16.424716 2601 raft_consensus.cc:2781] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:16.425029 2601 raft_consensus.cc:482] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:16.425299 2601 raft_consensus.cc:3037] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:16.448199 2601 raft_consensus.cc:504] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.450918 2348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e426200d0549cc8cab3b54553d69d0"
I20240502 14:06:16.451488 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "97b50f777bb6437aafe6fb971346ed39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9"
I20240502 14:06:16.451997 2490 raft_consensus.cc:3037] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:06:16.452265 2134 leader_election.cc:341] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:16.460979 2490 raft_consensus.cc:2454] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97b50f777bb6437aafe6fb971346ed39 in term 1.
I20240502 14:06:16.462137 2133 leader_election.cc:302] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8da730156fda48e7a89bbe894e2c31f9, 97b50f777bb6437aafe6fb971346ed39; no voters: 71e426200d0549cc8cab3b54553d69d0
I20240502 14:06:16.470765 2601 leader_election.cc:288] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [CANDIDATE]: Term 1 election: Requested vote from peers 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:16.471967 2601 raft_consensus.cc:2781] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:16.474179 2601 raft_consensus.cc:686] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [term 1 LEADER]: Becoming Leader. State: Replica: 97b50f777bb6437aafe6fb971346ed39, State: Running, Role: LEADER
I20240502 14:06:16.474962 2601 consensus_queue.cc:237] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.499096 2636 tablet_bootstrap.cc:492] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: No bootstrap required, opened a new log
I20240502 14:06:16.499706 2636 ts_tablet_manager.cc:1367] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Time spent bootstrapping tablet: real 0.107s user 0.013s sys 0.000s
I20240502 14:06:16.502713 2636 raft_consensus.cc:348] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.510893 2041 catalog_manager.cc:5484] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "97b50f777bb6437aafe6fb971346ed39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:16.520438 2636 raft_consensus.cc:374] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:16.534194 2636 raft_consensus.cc:729] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e426200d0549cc8cab3b54553d69d0, State: Initialized, Role: FOLLOWER
I20240502 14:06:16.536062 2636 consensus_queue.cc:260] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.545354 2660 ts_tablet_manager.cc:1888] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:16.565786 2636 ts_tablet_manager.cc:1398] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20240502 14:06:16.568113 2661 ts_tablet_manager.cc:1888] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:06:16.568552 2661 log.cc:1196] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/5305d46c75af4eab969a0e15778abc21
I20240502 14:06:16.569361 2661 ts_tablet_manager.cc:1909] T 5305d46c75af4eab969a0e15778abc21 P 71e426200d0549cc8cab3b54553d69d0: Deleting consensus metadata
I20240502 14:06:16.572304 2029 catalog_manager.cc:4830] TS 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): tablet 5305d46c75af4eab969a0e15778abc21 (table test-table [id=c2b9dc26492c4cd6af981d15ce632983]) successfully deleted
I20240502 14:06:16.570292 2660 log.cc:1196] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-0/wal/wals/5305d46c75af4eab969a0e15778abc21
I20240502 14:06:16.591327 2660 ts_tablet_manager.cc:1909] T 5305d46c75af4eab969a0e15778abc21 P 97b50f777bb6437aafe6fb971346ed39: Deleting consensus metadata
W20240502 14:06:16.611116 2028 catalog_manager.cc:4797] TS 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809): delete failed for tablet 5305d46c75af4eab969a0e15778abc21 because the tablet was not found. No further retry: Not found: Tablet not found: 5305d46c75af4eab969a0e15778abc21
I20240502 14:06:16.633293 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:16.598438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.27.174.65:32809)
0502 14:06:16.620486 (+ 22048us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:16.620512 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:16.620564 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:06:16.620720 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:16.620749 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:16.621054 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:16.622612 (+ 1558us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:16.622988 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.624523 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.628344 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.628836 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.628985 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:16.629242 (+ 257us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:16.631461 (+ 2219us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:16.631568 (+ 107us) server_negotiation.cc:299] Negotiation successful
0502 14:06:16.631740 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21469,"thread_start_us":22727,"threads_started":1}
I20240502 14:06:16.712862 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:16.679545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48131 (local address 127.27.174.65:32809)
0502 14:06:16.680554 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:16.680582 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:16.699585 (+ 19003us) server_negotiation.cc:407] Connection header received
0502 14:06:16.700418 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:16.700452 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:16.700734 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:16.700843 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:06:16.702709 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.703979 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.707576 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.708102 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.708746 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:06:16.708999 (+ 253us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:06:16.709099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:16.711269 (+ 2170us) server_negotiation.cc:299] Negotiation successful
0502 14:06:16.711438 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20240502 14:06:16.715728 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:16.678758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.27.174.65:48131)
0502 14:06:16.699397 (+ 20639us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:16.699459 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:16.700173 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:16.701367 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:16.701393 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:16.701450 (+ 57us) client_negotiation.cc:189] Negotiated authn=TOKEN
0502 14:06:16.702307 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:16.702357 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.704318 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:16.704350 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:16.705562 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:16.705611 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:16.708357 (+ 2746us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:16.708466 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0502 14:06:16.709220 (+ 754us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0502 14:06:16.709255 (+ 35us) client_negotiation.cc:769] Sending connection context
0502 14:06:16.713691 (+ 4436us) client_negotiation.cc:240] Negotiation successful
0502 14:06:16.713891 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19996,"thread_start_us":20796,"threads_started":1}
I20240502 14:06:16.755988 2490 raft_consensus.cc:1261] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Refusing update from remote peer 97b50f777bb6437aafe6fb971346ed39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:16.760798 2601 consensus_queue.cc:1035] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:06:16.888374 2133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.174.67:44793, user_credentials={real_user=slave}} blocked reactor thread for 48103.4us
I20240502 14:06:16.895646 2642 raft_consensus.cc:482] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:16.896075 2642 raft_consensus.cc:504] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97b50f777bb6437aafe6fb971346ed39" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 32809 } } peers { permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 } } peers { permanent_uuid: "8da730156fda48e7a89bbe894e2c31f9" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 44793 } }
I20240502 14:06:16.897879 2642 leader_election.cc:288] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809), 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793)
I20240502 14:06:16.898486 2348 raft_consensus.cc:3037] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:16.920688 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b50f777bb6437aafe6fb971346ed39" is_pre_election: true
I20240502 14:06:16.930670 2348 raft_consensus.cc:1261] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Refusing update from remote peer 97b50f777bb6437aafe6fb971346ed39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:16.934192 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473e73b7ee614ec1951b2bf0570da285" candidate_uuid: "71e426200d0549cc8cab3b54553d69d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da730156fda48e7a89bbe894e2c31f9" is_pre_election: true
I20240502 14:06:16.935650 2275 leader_election.cc:302] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71e426200d0549cc8cab3b54553d69d0; no voters: 8da730156fda48e7a89bbe894e2c31f9, 97b50f777bb6437aafe6fb971346ed39
I20240502 14:06:16.936374 2642 raft_consensus.cc:2726] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:16.938326 2666 consensus_queue.cc:1035] T 473e73b7ee614ec1951b2bf0570da285 P 97b50f777bb6437aafe6fb971346ed39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e426200d0549cc8cab3b54553d69d0" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:06:17.143283 2041 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:17.143867 2041 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:06:17.176043 2041 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 092ba44513f54368a9d741ec25b93335: Sending DeleteTablet for 3 replicas of tablet 473e73b7ee614ec1951b2bf0570da285
I20240502 14:06:17.178864 2187 tablet_service.cc:1514] Processing DeleteTablet for tablet 473e73b7ee614ec1951b2bf0570da285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:17 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:59332
I20240502 14:06:17.179700 2470 tablet_service.cc:1514] Processing DeleteTablet for tablet 473e73b7ee614ec1951b2bf0570da285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:17 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:45674
I20240502 14:06:17.180569 2328 tablet_service.cc:1514] Processing DeleteTablet for tablet 473e73b7ee614ec1951b2bf0570da285 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:17 UTC) from {username='oryx', principal='oryx/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54056
I20240502 14:06:17.181485 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2114
I20240502 14:06:17.186847 2684 tablet_replica.cc:330] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: stopping tablet replica
I20240502 14:06:17.187654 2684 raft_consensus.cc:2229] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:17.188856 2685 tablet_replica.cc:330] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: stopping tablet replica
I20240502 14:06:17.189741 2685 raft_consensus.cc:2229] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:17.190353 2685 raft_consensus.cc:2258] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:17.194483 2684 raft_consensus.cc:2258] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:17.195008 2685 ts_tablet_manager.cc:1875] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:17.197413 2684 ts_tablet_manager.cc:1875] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:17.212102 2685 ts_tablet_manager.cc:1888] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:06:17.212698 2685 log.cc:1196] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/473e73b7ee614ec1951b2bf0570da285
I20240502 14:06:17.213552 2685 ts_tablet_manager.cc:1909] T 473e73b7ee614ec1951b2bf0570da285 P 71e426200d0549cc8cab3b54553d69d0: Deleting consensus metadata
I20240502 14:06:17.216253 2029 catalog_manager.cc:4830] TS 71e426200d0549cc8cab3b54553d69d0 (127.27.174.66:41285): tablet 473e73b7ee614ec1951b2bf0570da285 (table test-table [id=c3e7f656e15347a2b0caa8e4af7ee64f]) successfully deleted
I20240502 14:06:17.233016 2684 ts_tablet_manager.cc:1888] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:06:17.233582 2684 log.cc:1196] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/473e73b7ee614ec1951b2bf0570da285
I20240502 14:06:17.234640 2684 ts_tablet_manager.cc:1909] T 473e73b7ee614ec1951b2bf0570da285 P 8da730156fda48e7a89bbe894e2c31f9: Deleting consensus metadata
W20240502 14:06:17.237627 2028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:32809 (error 108)
I20240502 14:06:17.245894 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2256
I20240502 14:06:17.254091 2028 catalog_manager.cc:4830] TS 8da730156fda48e7a89bbe894e2c31f9 (127.27.174.67:44793): tablet 473e73b7ee614ec1951b2bf0570da285 (table test-table [id=c3e7f656e15347a2b0caa8e4af7ee64f]) successfully deleted
I20240502 14:06:17.264240 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:17.255150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.0.0.1:59346)
0502 14:06:17.263272 (+ 8122us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:17.263550 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7317,"threads_started":1}
W20240502 14:06:17.265105 2028 catalog_manager.cc:4573] TS 97b50f777bb6437aafe6fb971346ed39 (127.27.174.65:32809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 473e73b7ee614ec1951b2bf0570da285: Network error: Client connection negotiation failed: client connection to 127.27.174.65:32809: connect: Connection refused (error 111)
I20240502 14:06:17.286657 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2398
I20240502 14:06:17.320371 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:17.319191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:32809 (local address 127.0.0.1:59352)
0502 14:06:17.319832 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:17.319985 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20240502 14:06:17.322958 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2008
2024-05-02T14:06:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (11360 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:06:17 dist-test-slave-dlb8 krb5kdc[2691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:06:17 dist-test-slave-dlb8 krb5kdc[2691](info): set up 2 sockets
May 02 14:06:17 dist-test-slave-dlb8 krb5kdc[2691](info): commencing operation
krb5kdc: starting...
W20240502 14:06:18.332243 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:06:18 dist-test-slave-dlb8 krb5kdc[2691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:06:18.536484 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43473
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:18.828610 2704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:18.829249 2704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:18.829830 2704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:18.859226 2704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:18.859565 2704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:18.859830 2704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:18.860046 2704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:18.860288 2704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:18.893931 2704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:18.895117 2704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:18.896227 2704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:18.912364 2711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:18.912405 2713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:18.912776 2710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:18.914002 2704 server_base.cc:1034] running on GCE node
I20240502 14:06:19.972568 2704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:19.975589 2704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:19.976989 2704 hybrid_clock.cc:648] HybridClock initialized: now 1714658779976953 us; error 53 us; skew 500 ppm
I20240502 14:06:19.977751 2704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-02T14:06:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (3653 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:06:21 dist-test-slave-dlb8 krb5kdc[2719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:06:21 dist-test-slave-dlb8 krb5kdc[2719](info): set up 2 sockets
May 02 14:06:21 dist-test-slave-dlb8 krb5kdc[2719](info): commencing operation
krb5kdc: starting...
W20240502 14:06:21.972179 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:06:22 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:06:22.169735 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41625
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:22.453889 2732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:22.454581 2732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:22.455152 2732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:22.483300 2732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:22.483585 2732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:22.483896 2732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:22.484127 2732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:22.484359 2732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:22.518083 2732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:22.519304 2732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:22.520282 2732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:22.536546 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:22.536599 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:22.538023 2732 server_base.cc:1034] running on GCE node
W20240502 14:06:22.536677 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:23.597793 2732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:23.602753 2732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:23.604110 2732 hybrid_clock.cc:648] HybridClock initialized: now 1714658783604085 us; error 50 us; skew 500 ppm
I20240502 14:06:23.604866 2732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:23 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658783, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:23.610623 2732 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:06:23.616266 2732 webserver.cc:460] Webserver started at http://127.27.174.126:39659/ using document root <none> and password file <none>
I20240502 14:06:23.617173 2732 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:23.617386 2732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:23.617817 2732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:23.622910 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "7e4b84b7731f4e77a2a6a9f82d141588"
format_stamp: "Formatted at 2024-05-02 14:06:23 on dist-test-slave-dlb8"
server_key: "3c87b2737563021bce26a7f0a35e872f"
server_key_iv: "76785a1b482940893b882c3dffb01551"
server_key_version: "encryptionkey@0"
I20240502 14:06:23.623999 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "7e4b84b7731f4e77a2a6a9f82d141588"
format_stamp: "Formatted at 2024-05-02 14:06:23 on dist-test-slave-dlb8"
server_key: "3c87b2737563021bce26a7f0a35e872f"
server_key_iv: "76785a1b482940893b882c3dffb01551"
server_key_version: "encryptionkey@0"
I20240502 14:06:23.630985 2732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:06:23.636298 2749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:23.637295 2732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:06:23.637626 2732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "7e4b84b7731f4e77a2a6a9f82d141588"
format_stamp: "Formatted at 2024-05-02 14:06:23 on dist-test-slave-dlb8"
server_key: "3c87b2737563021bce26a7f0a35e872f"
server_key_iv: "76785a1b482940893b882c3dffb01551"
server_key_version: "encryptionkey@0"
I20240502 14:06:23.637990 2732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:23.694291 2732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:23.699385 2732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:23.699846 2732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:23.765626 2732 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41625
I20240502 14:06:23.765725 2800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41625 every 8 connection(s)
I20240502 14:06:23.768205 2732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:06:23.773075 2801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:23.776414 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2732
I20240502 14:06:23.776921 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:06:23.778548 28345 external_mini_cluster.cc:1428] Setting key 16ad98595f492831e40c8dda8974ad05
I20240502 14:06:23.796547 2801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Bootstrap starting.
May 02 14:06:23 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:23.803198 2801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:23.804857 2801 log.cc:824] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:23.810756 2801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: No bootstrap required, opened a new log
I20240502 14:06:23.814671 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:23.782236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.27.174.126:41625)
0502 14:06:23.784982 (+ 2746us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:23.785007 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:23.785054 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:06:23.789006 (+ 3952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:23.789052 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:23.789373 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:23.789700 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:23.793244 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:23.794645 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:23.796559 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:23.797037 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:23.803100 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:23.803199 (+ 99us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:23.803226 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:23.803321 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:23.808178 (+ 4857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:23.809682 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:23.809704 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:23.809734 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:23.809899 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:23.810539 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:23.810557 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:23.810570 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:23.811176 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:23.811506 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:23.812240 (+ 734us) server_negotiation.cc:299] Negotiation successful
0502 14:06:23.812661 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2256,"thread_start_us":1742,"threads_started":1}
I20240502 14:06:23.831176 2801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:23.831884 2801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:23.832113 2801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e4b84b7731f4e77a2a6a9f82d141588, State: Initialized, Role: FOLLOWER
I20240502 14:06:23.832795 2801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:23.833271 2801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:23.833587 2801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:23.833930 2801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:23.837898 2801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:23.838624 2801 leader_election.cc:302] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e4b84b7731f4e77a2a6a9f82d141588; no voters:
I20240502 14:06:23.840195 2801 leader_election.cc:288] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:23.840569 2806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:23.843499 2806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 LEADER]: Becoming Leader. State: Replica: 7e4b84b7731f4e77a2a6a9f82d141588, State: Running, Role: LEADER
I20240502 14:06:23.844274 2806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:23.844897 2801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:23.854092 2807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } } }
I20240502 14:06:23.854967 2807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:23.855240 2808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e4b84b7731f4e77a2a6a9f82d141588. Latest consensus state: current_term: 1 leader_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } } }
I20240502 14:06:23.856657 2808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:23.858861 2815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:23.870250 2815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:23.884733 2815 catalog_manager.cc:1347] Generated new cluster ID: 197a5892534e465b8aa55721399f04f5
I20240502 14:06:23.885027 2815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:23.901985 2815 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:23.903895 2815 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:23.916849 2815 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Generated new TSK 0
I20240502 14:06:23.917793 2815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:06:23.980155 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:24.259802 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:24.260270 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:24.260810 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:24.288928 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:24.289290 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:24.290139 2827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:06:24.323120 2827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:24.324481 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:24.325465 2827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:24.343421 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:24.344142 2834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:24.345397 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:25.766976 2835 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1420 milliseconds
I20240502 14:06:25.767109 2827 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:06:25.768486 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:25.771265 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:25.772789 2827 hybrid_clock.cc:648] HybridClock initialized: now 1714658785772725 us; error 70 us; skew 500 ppm
I20240502 14:06:25.773710 2827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:25 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658785, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:25.785485 2827 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:06:25.791167 2827 webserver.cc:460] Webserver started at http://127.27.174.65:36135/ using document root <none> and password file <none>
I20240502 14:06:25.791994 2827 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:25.792192 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:25.792603 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:25.797137 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "b5301da9b8454745a3139f55505b6371"
format_stamp: "Formatted at 2024-05-02 14:06:25 on dist-test-slave-dlb8"
server_key: "d77f19ffa5c79988d7a6307a455601a7"
server_key_iv: "355bcdb8cc181971d45760be4c61c9b1"
server_key_version: "encryptionkey@0"
I20240502 14:06:25.798307 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "b5301da9b8454745a3139f55505b6371"
format_stamp: "Formatted at 2024-05-02 14:06:25 on dist-test-slave-dlb8"
server_key: "d77f19ffa5c79988d7a6307a455601a7"
server_key_iv: "355bcdb8cc181971d45760be4c61c9b1"
server_key_version: "encryptionkey@0"
I20240502 14:06:25.804976 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:25.810389 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:25.811358 2827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:06:25.811648 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "b5301da9b8454745a3139f55505b6371"
format_stamp: "Formatted at 2024-05-02 14:06:25 on dist-test-slave-dlb8"
server_key: "d77f19ffa5c79988d7a6307a455601a7"
server_key_iv: "355bcdb8cc181971d45760be4c61c9b1"
server_key_version: "encryptionkey@0"
I20240502 14:06:25.812001 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:25.870973 2827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:25.876053 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:25.876510 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:25.878955 2827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:25.882773 2827 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:25.882979 2827 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:25.883189 2827 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:25.883337 2827 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:26.007012 2827 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40915
I20240502 14:06:26.007084 2956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40915 every 8 connection(s)
I20240502 14:06:26.009537 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:06:26.020035 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2827
I20240502 14:06:26.020422 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:06:26.021602 28345 external_mini_cluster.cc:1428] Setting key fd5533d58fedb3a2fd8c1a506f7c2b8d
May 02 14:06:26 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658785, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:26.039866 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:26.013484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35733 (local address 127.27.174.126:41625)
0502 14:06:26.014632 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:26.014646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:26.016868 (+ 2222us) server_negotiation.cc:407] Connection header received
0502 14:06:26.021167 (+ 4299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:26.021196 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:26.021416 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:26.021615 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:26.023841 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:26.024716 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:26.025970 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:26.026495 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:26.030027 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:26.030129 (+ 102us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:26.030146 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:26.030263 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:26.033998 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:26.035030 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:26.035054 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:26.035067 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:26.035189 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:26.035730 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:26.035750 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:26.035763 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:26.036107 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:26.036353 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:26.037294 (+ 941us) server_negotiation.cc:299] Negotiation successful
0502 14:06:26.037755 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":484,"threads_started":1}
I20240502 14:06:26.040447 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:26.013766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:35733)
0502 14:06:26.016586 (+ 2820us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:26.016649 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:26.020846 (+ 4197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:26.021812 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:26.021848 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:26.022654 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:26.023553 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:26.023601 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:26.024928 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:26.024950 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:26.025762 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:26.025794 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:26.026143 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:26.027050 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:26.027107 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:26.029783 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:26.034228 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:26.034259 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:26.034280 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:26.034854 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:26.035352 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:26.035371 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:26.035385 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:26.035563 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:26.036306 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:26.036335 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:26.036728 (+ 393us) client_negotiation.cc:769] Sending connection context
0502 14:06:26.037042 (+ 314us) client_negotiation.cc:240] Negotiation successful
0502 14:06:26.037623 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"thread_start_us":1557,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20240502 14:06:26.042505 2957 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41625
I20240502 14:06:26.042858 2957 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:26.043910 2957 heartbeater.cc:511] Master 127.27.174.126:41625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:06:26.046656 2766 ts_manager.cc:194] Registered new tserver with Master: b5301da9b8454745a3139f55505b6371 (127.27.174.65:40915)
I20240502 14:06:26.048929 2766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:35733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:06:26.069890 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:26.355832 2963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:26.356309 2963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:26.356884 2963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:26.385852 2963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:26.386214 2963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:26.387032 2963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:06:26.420120 2963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:26.421244 2963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:26.422319 2963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:26.437608 2970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:27.052488 2957 heartbeater.cc:503] Master 127.27.174.126:41625 was elected leader, sending a full tablet report...
W20240502 14:06:26.437690 2969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:26.439760 2963 server_base.cc:1034] running on GCE node
W20240502 14:06:26.439049 2972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:27.482861 2963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:27.485533 2963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:27.486925 2963 hybrid_clock.cc:648] HybridClock initialized: now 1714658787486885 us; error 58 us; skew 500 ppm
I20240502 14:06:27.487680 2963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:27 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658787, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:27.493289 2963 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:06:27.498497 2963 webserver.cc:460] Webserver started at http://127.27.174.66:42093/ using document root <none> and password file <none>
I20240502 14:06:27.499300 2963 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:27.499466 2963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:27.499837 2963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:27.504601 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "d7df23f509b34ee88e93556ec721657f"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-dlb8"
server_key: "536250a0293c0107fd98fecc5d47d93a"
server_key_iv: "f07ae9d685fd004fedf2a299b080d5ec"
server_key_version: "encryptionkey@0"
I20240502 14:06:27.505642 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "d7df23f509b34ee88e93556ec721657f"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-dlb8"
server_key: "536250a0293c0107fd98fecc5d47d93a"
server_key_iv: "f07ae9d685fd004fedf2a299b080d5ec"
server_key_version: "encryptionkey@0"
I20240502 14:06:27.512173 2963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:27.517434 2981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:27.518486 2963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:06:27.518810 2963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "d7df23f509b34ee88e93556ec721657f"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-dlb8"
server_key: "536250a0293c0107fd98fecc5d47d93a"
server_key_iv: "f07ae9d685fd004fedf2a299b080d5ec"
server_key_version: "encryptionkey@0"
I20240502 14:06:27.519120 2963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:27.575058 2963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:27.580315 2963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:27.580729 2963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:27.583103 2963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:27.586687 2963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:27.586867 2963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:27.587111 2963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:27.587253 2963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:27.711650 2963 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41505
I20240502 14:06:27.711748 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41505 every 8 connection(s)
I20240502 14:06:27.714080 2963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:06:27.722770 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 2963
I20240502 14:06:27.723568 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:06:27.725091 28345 external_mini_cluster.cc:1428] Setting key 79487a8a03162b2dd7b2d4e6776df310
May 02 14:06:27 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658787, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:27.749859 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:27.718217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:54003 (local address 127.27.174.126:41625)
0502 14:06:27.719440 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:27.719455 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:27.721584 (+ 2129us) server_negotiation.cc:407] Connection header received
0502 14:06:27.727867 (+ 6283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:27.727894 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:27.728102 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:27.728285 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:27.731022 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:27.731959 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:27.733883 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:27.734281 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:27.739549 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:27.739657 (+ 108us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:27.739688 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:27.739841 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:27.743690 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:27.745337 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:27.745363 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:27.745376 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:27.745526 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:27.746298 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:27.746319 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:27.746350 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:27.746757 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:27.746936 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:27.748222 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0502 14:06:27.748556 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":480,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:06:27.751546 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:27.718442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:54003)
0502 14:06:27.721387 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:27.721450 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:27.727552 (+ 6102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:27.728487 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:27.728510 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:27.729360 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:27.730706 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:27.730790 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:27.732235 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:27.732276 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:27.733653 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:27.733710 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:27.734139 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:27.735108 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:27.735217 (+ 109us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:27.739277 (+ 4060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:27.743916 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:27.743956 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:27.743996 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:27.745079 (+ 1083us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:27.745726 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:27.745756 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:27.745782 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:27.746102 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:27.746959 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:27.747003 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:27.747621 (+ 618us) client_negotiation.cc:769] Sending connection context
0502 14:06:27.748075 (+ 454us) client_negotiation.cc:240] Negotiation successful
0502 14:06:27.748720 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2293,"thread_start_us":1719,"threads_started":1}
I20240502 14:06:27.753291 3094 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:06:27.753629 3094 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:27.754870 3094 heartbeater.cc:511] Master 127.27.174.126:41625 requested a full tablet report, sending...
I20240502 14:06:27.757258 2766 ts_manager.cc:194] Registered new tserver with Master: d7df23f509b34ee88e93556ec721657f (127.27.174.66:41505)
I20240502 14:06:27.758512 2766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:54003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:06:27.778935 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:28.062044 3100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:28.062558 3100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:28.063150 3100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:28.091835 3100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:28.092144 3100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:28.093055 3100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:06:28.126111 3100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:28.127213 3100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:28.128199 3100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:28.143721 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:28.761394 3094 heartbeater.cc:503] Master 127.27.174.126:41625 was elected leader, sending a full tablet report...
W20240502 14:06:28.144340 3109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.144785 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:28.145303 3100 server_base.cc:1034] running on GCE node
I20240502 14:06:29.184991 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:29.187201 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:29.188534 3100 hybrid_clock.cc:648] HybridClock initialized: now 1714658789188500 us; error 56 us; skew 500 ppm
I20240502 14:06:29.189312 3100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:29 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:29.194656 3100 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:06:29.198935 3100 webserver.cc:460] Webserver started at http://127.27.174.67:44889/ using document root <none> and password file <none>
I20240502 14:06:29.199717 3100 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:29.199899 3100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:29.200315 3100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:29.205067 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "95325d2dad794d00bbe39a78d63a88ed"
format_stamp: "Formatted at 2024-05-02 14:06:29 on dist-test-slave-dlb8"
server_key: "c5babab17cd0fc3b069876b3a598b2cc"
server_key_iv: "38869cd338a617f04f5b743f5a5725b6"
server_key_version: "encryptionkey@0"
I20240502 14:06:29.206137 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "95325d2dad794d00bbe39a78d63a88ed"
format_stamp: "Formatted at 2024-05-02 14:06:29 on dist-test-slave-dlb8"
server_key: "c5babab17cd0fc3b069876b3a598b2cc"
server_key_iv: "38869cd338a617f04f5b743f5a5725b6"
server_key_version: "encryptionkey@0"
I20240502 14:06:29.213032 3100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:06:29.218309 3117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:29.219312 3100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:06:29.219642 3100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "95325d2dad794d00bbe39a78d63a88ed"
format_stamp: "Formatted at 2024-05-02 14:06:29 on dist-test-slave-dlb8"
server_key: "c5babab17cd0fc3b069876b3a598b2cc"
server_key_iv: "38869cd338a617f04f5b743f5a5725b6"
server_key_version: "encryptionkey@0"
I20240502 14:06:29.219959 3100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:29.269559 3100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:29.274554 3100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:29.274953 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:29.277179 3100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:29.281075 3100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:29.281244 3100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:29.281484 3100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:29.281618 3100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:29.405632 3100 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41781
I20240502 14:06:29.405730 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41781 every 8 connection(s)
I20240502 14:06:29.408126 3100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:06:29.412770 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3100
I20240502 14:06:29.413388 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:06:29.414908 28345 external_mini_cluster.cc:1428] Setting key ef90909b56fad6112cb25c998fb298e6
May 02 14:06:29 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:06:29.438838 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:29.412066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52205 (local address 127.27.174.126:41625)
0502 14:06:29.413587 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:29.413601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:29.415060 (+ 1459us) server_negotiation.cc:407] Connection header received
0502 14:06:29.419999 (+ 4939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:29.420032 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:29.420192 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:29.420334 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:29.422682 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:29.423630 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:29.424933 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:29.425510 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:29.429135 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:29.429200 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:29.429219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:29.429309 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:29.432811 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:29.433840 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:29.433864 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:29.433880 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:29.434010 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:29.434610 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:29.434629 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:29.434642 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:29.434995 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:29.435206 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:29.436085 (+ 879us) server_negotiation.cc:299] Negotiation successful
0502 14:06:29.436523 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":849,"threads_started":1}
I20240502 14:06:29.439394 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:29.412392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:52205)
0502 14:06:29.414830 (+ 2438us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:29.414892 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:29.419700 (+ 4808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:29.420610 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:29.420643 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:29.421390 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:29.422421 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:29.422466 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:29.423855 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:29.423877 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:29.424717 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:29.424752 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:29.425159 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:29.426031 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:29.426134 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:29.428885 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:29.433013 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:29.433042 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:29.433063 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:29.433655 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:29.434205 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:06:29.434232 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:06:29.434246 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:06:29.434427 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:06:29.435175 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:29.435205 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:29.435579 (+ 374us) client_negotiation.cc:769] Sending connection context
0502 14:06:29.435874 (+ 295us) client_negotiation.cc:240] Negotiation successful
0502 14:06:29.436436 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"thread_start_us":1420,"threads_started":1}
I20240502 14:06:29.440922 3230 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41625
I20240502 14:06:29.441287 3230 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:29.442274 3230 heartbeater.cc:511] Master 127.27.174.126:41625 requested a full tablet report, sending...
I20240502 14:06:29.444340 2766 ts_manager.cc:194] Registered new tserver with Master: 95325d2dad794d00bbe39a78d63a88ed (127.27.174.67:41781)
I20240502 14:06:29.445806 2766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:52205
I20240502 14:06:29.446959 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
Entry for principal oryx/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab.
WARNING: no policy specified for oryx/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
Entry for principal oryx/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab.
WARNING: no policy specified for oryx/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
Entry for principal oryx/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab.
I20240502 14:06:29.577443 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2827
I20240502 14:06:29.599305 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2963
I20240502 14:06:29.622025 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3100
I20240502 14:06:29.644362 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 2732
I20240502 14:06:29.675040 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41625
--webserver_interface=127.27.174.126
--webserver_port=39659
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:06:29.956491 3240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:29.957032 3240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:29.957564 3240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:29.984855 3240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:29.985137 3240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:29.985453 3240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:29.985694 3240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:29.985930 3240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:30.018064 3240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=39659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:30.019201 3240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:30.020187 3240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:30.035782 3246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:30.035892 3249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:30.037338 3240 server_base.cc:1034] running on GCE node
W20240502 14:06:30.036240 3247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:31.100891 3240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:31.104089 3240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:31.105526 3240 hybrid_clock.cc:648] HybridClock initialized: now 1714658791105500 us; error 50 us; skew 500 ppm
I20240502 14:06:31.106377 3240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:31 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658791, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:31.119792 3240 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:06:31.124914 3240 webserver.cc:460] Webserver started at http://127.27.174.126:39659/ using document root <none> and password file <none>
I20240502 14:06:31.125783 3240 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:31.126004 3240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:31.133994 3240 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 14:06:31.138296 3257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:31.139282 3240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:06:31.139709 3240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "7e4b84b7731f4e77a2a6a9f82d141588"
format_stamp: "Formatted at 2024-05-02 14:06:23 on dist-test-slave-dlb8"
server_key: "3c87b2737563021bce26a7f0a35e872f"
server_key_iv: "76785a1b482940893b882c3dffb01551"
server_key_version: "encryptionkey@0"
I20240502 14:06:31.142024 3240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:31.200850 3240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:31.206251 3240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:31.206710 3240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:31.271565 3240 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41625
I20240502 14:06:31.271708 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41625 every 8 connection(s)
I20240502 14:06:31.274293 3240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:06:31.279686 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3240
I20240502 14:06:31.282002 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:40915
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:41625
--webserver_port=36135
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.65
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240502 14:06:31.287115 3309 sys_catalog.cc:263] Verifying existing consensus state
I20240502 14:06:31.292860 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Bootstrap starting.
I20240502 14:06:31.327764 3309 log.cc:824] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:31.343441 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.344206 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Bootstrap complete.
I20240502 14:06:31.366310 3309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:31.368505 3309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e4b84b7731f4e77a2a6a9f82d141588, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.369310 3309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:31.369774 3309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:31.370085 3309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:31.370370 3309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:31.376195 3309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:31.376888 3309 leader_election.cc:302] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e4b84b7731f4e77a2a6a9f82d141588; no voters:
I20240502 14:06:31.379029 3309 leader_election.cc:288] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 14:06:31.379421 3313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:31.382406 3313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [term 2 LEADER]: Becoming Leader. State: Replica: 7e4b84b7731f4e77a2a6a9f82d141588, State: Running, Role: LEADER
I20240502 14:06:31.383234 3313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } }
I20240502 14:06:31.384027 3309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:31.393409 3314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } } }
I20240502 14:06:31.393415 3315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e4b84b7731f4e77a2a6a9f82d141588. Latest consensus state: current_term: 2 leader_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4b84b7731f4e77a2a6a9f82d141588" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41625 } } }
I20240502 14:06:31.394253 3314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:31.394308 3315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:31.406630 3320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:31.418287 3320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:31.422644 3320 catalog_manager.cc:1259] Loaded cluster ID: 197a5892534e465b8aa55721399f04f5
I20240502 14:06:31.422981 3320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:31.430078 3320 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:31.434329 3320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e4b84b7731f4e77a2a6a9f82d141588: Loaded TSK: 0
I20240502 14:06:31.435846 3320 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240502 14:06:31.607631 3311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:31.608201 3311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:31.608875 3311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:31.639251 3311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:31.639600 3311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:31.640473 3311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:06:31.674752 3311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.65.keytab
--principal=oryx/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:40915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=36135
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:31.675884 3311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:31.676872 3311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:31.694031 3339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:31.695811 3336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:31.694090 3337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:32.984711 3338 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1290 milliseconds
I20240502 14:06:32.984844 3311 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:06:32.986164 3311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:32.988845 3311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:32.990337 3311 hybrid_clock.cc:648] HybridClock initialized: now 1714658792990279 us; error 62 us; skew 500 ppm
I20240502 14:06:32.991335 3311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:32 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658792, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:32.998389 3311 init.cc:377] Logged in from keytab as oryx/127.27.174.65@KRBTEST.COM (short username oryx)
I20240502 14:06:33.003724 3311 webserver.cc:460] Webserver started at http://127.27.174.65:36135/ using document root <none> and password file <none>
I20240502 14:06:33.004797 3311 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:33.005075 3311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:33.014432 3311 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 14:06:33.018376 3347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:33.019447 3311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:06:33.019726 3311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "b5301da9b8454745a3139f55505b6371"
format_stamp: "Formatted at 2024-05-02 14:06:25 on dist-test-slave-dlb8"
server_key: "d77f19ffa5c79988d7a6307a455601a7"
server_key_iv: "355bcdb8cc181971d45760be4c61c9b1"
server_key_version: "encryptionkey@0"
I20240502 14:06:33.021950 3311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:33.072156 3311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:33.077523 3311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:33.077942 3311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:33.080559 3311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:33.085053 3311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:33.085291 3311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:33.085515 3311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:33.085664 3311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:33.248522 3311 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40915
I20240502 14:06:33.248683 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40915 every 8 connection(s)
I20240502 14:06:33.251212 3311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:06:33.256705 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3311
I20240502 14:06:33.259536 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:41505
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:41625
--webserver_port=42093
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.66
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:33.295801 3462 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:33.255813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42239 (local address 127.27.174.126:41625)
0502 14:06:33.267405 (+ 11592us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:33.267442 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:33.268342 (+ 900us) server_negotiation.cc:407] Connection header received
0502 14:06:33.276359 (+ 8017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:33.276398 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:33.276825 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:33.277174 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:33.280716 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.282976 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.284985 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.285556 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.294219 (+ 8663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:42239: BlockingRecv error: recv got EOF from 127.27.174.65:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":10840,"thread_start_us":10359,"threads_started":1}
I20240502 14:06:33.295830 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:33.255838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:42239)
0502 14:06:33.268054 (+ 12216us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:33.268147 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:33.275658 (+ 7511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:33.278042 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:33.278100 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:33.279191 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:33.280365 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.280433 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.283304 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.283332 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:33.284682 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.284732 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.286251 (+ 1519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.287241 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:33.287347 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:33.293690 (+ 6343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11518,"thread_start_us":16820,"threads_started":1}
W20240502 14:06:33.296919 3460 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41625 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41625: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:33.311743 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:33.298411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:49119)
0502 14:06:33.299740 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:33.299782 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:33.300004 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:33.300708 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:33.300733 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:33.301365 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:33.301884 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.301928 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.303557 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.303584 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:33.304683 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.304731 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.305030 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.305879 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:33.305958 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:33.310358 (+ 4400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":792}
W20240502 14:06:33.311811 3462 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:33.298507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49119 (local address 127.27.174.126:41625)
0502 14:06:33.299148 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:33.299172 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:33.299903 (+ 731us) server_negotiation.cc:407] Connection header received
0502 14:06:33.300209 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:33.300236 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:33.300448 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:33.300616 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:33.302166 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.303342 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.304936 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.305449 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.310717 (+ 5268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:49119: BlockingRecv error: recv got EOF from 127.27.174.65:49119 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:33 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:33.326167 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:33.313354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:58597)
0502 14:06:33.314204 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:33.314244 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:33.314458 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:33.315170 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:33.315196 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:33.315859 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:33.316310 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.316356 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.317997 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.318044 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:33.319183 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:33.319245 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.319537 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.320324 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:33.320413 (+ 89us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:33.324769 (+ 4356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280}
W20240502 14:06:33.326221 3462 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:33.313610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:58597 (local address 127.27.174.126:41625)
0502 14:06:33.314384 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:33.314412 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:33.314454 (+ 42us) server_negotiation.cc:407] Connection header received
0502 14:06:33.314679 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:33.314709 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:33.314917 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:33.315086 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:33.316580 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.317779 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:33.319450 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:33.319922 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:33.325124 (+ 5202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:58597: BlockingRecv error: recv got EOF from 127.27.174.65:58597 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20240502 14:06:33.326983 3460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240502 14:06:33.571604 3464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:33.572084 3464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:33.572691 3464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:33.602910 3464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:33.603271 3464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:33.604144 3464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:06:33.639464 3464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.66.keytab
--principal=oryx/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:41505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=42093
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:33.640578 3464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:33.641585 3464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:33.659453 3473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
May 02 14:06:34 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:34 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:34.341404 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:34.328383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:46781)
0502 14:06:34.330558 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:34.330591 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:34.330852 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:34.331492 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:34.331520 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:34.332164 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:34.332722 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:34.332756 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.334118 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:34.334140 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:34.334919 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:34.334954 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.335263 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:34.336333 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:34.336391 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:34.339792 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":1483,"threads_started":1}
W20240502 14:06:34.341492 3476 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:34.328328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:46781 (local address 127.27.174.126:41625)
0502 14:06:34.329952 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:34.329972 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:34.330737 (+ 765us) server_negotiation.cc:407] Connection header received
0502 14:06:34.331053 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:34.331070 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:34.331236 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:34.331384 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:34.332960 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.333843 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:34.335140 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.335730 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:34.340182 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:46781: BlockingRecv error: recv got EOF from 127.27.174.65:46781 (error 108)
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":624,"threads_started":1}
W20240502 14:06:33.661388 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:33.659483 3471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:33.661481 3464 server_base.cc:1034] running on GCE node
I20240502 14:06:34.730623 3464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:34.732916 3464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:34.734251 3464 hybrid_clock.cc:648] HybridClock initialized: now 1714658794734195 us; error 68 us; skew 500 ppm
I20240502 14:06:34.735026 3464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:34 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658794, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:34.740774 3464 init.cc:377] Logged in from keytab as oryx/127.27.174.66@KRBTEST.COM (short username oryx)
I20240502 14:06:34.745810 3464 webserver.cc:460] Webserver started at http://127.27.174.66:42093/ using document root <none> and password file <none>
I20240502 14:06:34.746685 3464 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:34.746881 3464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:34.754704 3464 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 14:06:34.758648 3483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:34.759617 3464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:06:34.759913 3464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "d7df23f509b34ee88e93556ec721657f"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-dlb8"
server_key: "536250a0293c0107fd98fecc5d47d93a"
server_key_iv: "f07ae9d685fd004fedf2a299b080d5ec"
server_key_version: "encryptionkey@0"
I20240502 14:06:34.762194 3464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:34.840559 3464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:34.846601 3464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:34.847033 3464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:34.849493 3464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:34.853281 3464 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:34.853488 3464 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:34.853739 3464 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:34.853927 3464 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:34.978061 3464 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41505
I20240502 14:06:34.978196 3595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41505 every 8 connection(s)
I20240502 14:06:34.980645 3464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:06:34.986235 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3464
I20240502 14:06:34.988072 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:41781
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:41625
--webserver_port=44889
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.174.67
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:06:34 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:34 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:35.006317 3598 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:34.984556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44675 (local address 127.27.174.126:41625)
0502 14:06:34.986060 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:34.986092 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:34.987486 (+ 1394us) server_negotiation.cc:407] Connection header received
0502 14:06:34.992078 (+ 4592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:34.992103 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:34.992287 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:34.992443 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:34.995256 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.996702 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:34.998013 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.998671 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.005464 (+ 6793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:44675: BlockingRecv error: recv got EOF from 127.27.174.66:44675 (error 108)
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":608,"threads_started":1}
I20240502 14:06:35.007573 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:34.984860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:44675)
0502 14:06:34.987275 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:34.987337 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:34.991737 (+ 4400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:34.992741 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:34.992765 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:34.993662 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:34.994982 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:34.995025 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.996948 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:34.996972 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:34.997772 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:34.997812 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:34.998124 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:34.998968 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:34.999034 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:35.005148 (+ 6114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1913,"thread_start_us":1427,"threads_started":1}
W20240502 14:06:35.008571 3596 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41625 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41625: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:35.019152 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:35.009594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:44165)
0502 14:06:35.010117 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:35.010147 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:35.010295 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:35.011105 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:35.011121 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:35.011565 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:35.011881 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.011912 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.013520 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.013538 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:35.014270 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.014305 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.014504 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.015140 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:35.015198 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:35.018282 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:06:35.019873 3598 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:35.009805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:44165 (local address 127.27.174.126:41625)
0502 14:06:35.010498 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:35.010522 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:35.010560 (+ 38us) server_negotiation.cc:407] Connection header received
0502 14:06:35.010677 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:35.010702 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:35.010903 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:35.011069 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:35.012089 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.013319 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.014466 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.014961 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.018686 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:44165: BlockingRecv error: recv got EOF from 127.27.174.66:44165 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:35.029632 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:35.020521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:41693)
0502 14:06:35.021047 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:35.021076 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:35.021227 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:35.021855 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:35.021874 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:35.022366 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:35.022715 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.022749 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.024349 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.024368 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:35.024969 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.024996 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.025190 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.025790 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:35.025845 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:35.028743 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20240502 14:06:35.030222 3596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240502 14:06:35.030278 3598 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:35.020508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:41693 (local address 127.27.174.126:41625)
0502 14:06:35.021111 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:35.021132 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:35.021174 (+ 42us) server_negotiation.cc:407] Connection header received
0502 14:06:35.021395 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:35.021418 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:35.021613 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:35.021780 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:35.022937 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.024089 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.025164 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.025632 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.029122 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:41693: BlockingRecv error: recv got EOF from 127.27.174.66:41693 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20240502 14:06:35.293408 3600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:35.293852 3600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:35.294418 3600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:35.323506 3600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:35.323820 3600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:35.324644 3600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:35 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:35.356089 3598 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:35.343907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:59337 (local address 127.27.174.126:41625)
0502 14:06:35.344586 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:35.344612 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:35.345590 (+ 978us) server_negotiation.cc:407] Connection header received
0502 14:06:35.345842 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:35.345867 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:35.346094 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:35.346267 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:35.348041 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.349369 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.350595 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.351108 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.355395 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:59337: BlockingRecv error: recv got EOF from 127.27.174.65:59337 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20240502 14:06:35.356523 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:35.343529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:59337)
0502 14:06:35.345432 (+ 1903us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:35.345474 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:35.345662 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:35.346329 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:35.346352 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:35.347185 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:35.347781 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.347828 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.349608 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:35.349631 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:35.350379 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:35.350414 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:35.350631 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:35.351329 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:35.351382 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:35.355141 (+ 3759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":1244,"threads_started":1}
I20240502 14:06:35.363332 3600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/krb5kdc/oryx_127.27.174.67.keytab
--principal=oryx/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:41781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=44889
--tserver_master_addrs=127.27.174.126:41625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:35.364483 3600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:35.365458 3600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:35.380120 3607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:36.048549 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.031419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:39663)
0502 14:06:36.034064 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:36.034149 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:36.034533 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:36.035267 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:36.035309 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:36.036604 (+ 1295us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:36.037684 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.037763 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.039396 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.039463 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:36.041229 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.041299 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.041793 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.042812 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:36.042935 (+ 123us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:36.047585 (+ 4650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1717,"thread_start_us":1311,"threads_started":1}
W20240502 14:06:36.050313 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:36.031823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39663 (local address 127.27.174.126:41625)
0502 14:06:36.034221 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.034239 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.034281 (+ 42us) server_negotiation.cc:407] Connection header received
0502 14:06:36.034792 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.034807 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.034958 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.035085 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.038149 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.039057 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.041566 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.041904 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.048256 (+ 6352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:39663: BlockingRecv error: recv got EOF from 127.27.174.66:39663 (error 108)
Metrics: {"server-negotiator.queue_time_us":1966,"thread_start_us":1679,"threads_started":1}
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:36.375680 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:36.358367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:44023 (local address 127.27.174.126:41625)
0502 14:06:36.359196 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.359222 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.360094 (+ 872us) server_negotiation.cc:407] Connection header received
0502 14:06:36.360396 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.360415 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.360576 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.360706 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.362123 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.362935 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.364037 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.364576 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.374564 (+ 9988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:44023: BlockingRecv error: recv got EOF from 127.27.174.65:44023 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20240502 14:06:36.375669 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.358275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:44023)
0502 14:06:36.359938 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:36.359972 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:36.360197 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:36.360795 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:36.360817 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:36.361405 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:36.361823 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.361862 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.363151 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.363173 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:36.363824 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.363856 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.364109 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.370774 (+ 6665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:36.370829 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:36.374190 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":1043,"threads_started":1}
W20240502 14:06:35.380746 3608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:35.381695 3610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:36.425973 3609 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:06:36.426029 3600 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:06:36.429692 3600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:36.432390 3600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:36.433795 3600 hybrid_clock.cc:648] HybridClock initialized: now 1714658796433771 us; error 48 us; skew 500 ppm
I20240502 14:06:36.434562 3600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:06:36.440116 3600 init.cc:377] Logged in from keytab as oryx/127.27.174.67@KRBTEST.COM (short username oryx)
I20240502 14:06:36.445617 3600 webserver.cc:460] Webserver started at http://127.27.174.67:44889/ using document root <none> and password file <none>
I20240502 14:06:36.446579 3600 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:36.446808 3600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:36.454809 3600 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 14:06:36.458757 3621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:36.459834 3600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:06:36.460129 3600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "95325d2dad794d00bbe39a78d63a88ed"
format_stamp: "Formatted at 2024-05-02 14:06:29 on dist-test-slave-dlb8"
server_key: "c5babab17cd0fc3b069876b3a598b2cc"
server_key_iv: "38869cd338a617f04f5b743f5a5725b6"
server_key_version: "encryptionkey@0"
I20240502 14:06:36.462461 3600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:36.515380 3600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:36.520752 3600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:36.521224 3600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:36.523635 3600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:36.527988 3600 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:36.528180 3600 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:36.528476 3600 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:36.528633 3600 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:36.658330 3600 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41781
I20240502 14:06:36.658437 3733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41781 every 8 connection(s)
I20240502 14:06:36.660979 3600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:06:36.664228 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3600
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:36.690135 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:36.665827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:38075 (local address 127.27.174.126:41625)
0502 14:06:36.666511 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.666537 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.669423 (+ 2886us) server_negotiation.cc:407] Connection header received
0502 14:06:36.676222 (+ 6799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.676243 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.676406 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.676554 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.678909 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.679845 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.681129 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.681664 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.688930 (+ 7266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:38075: BlockingRecv error: recv got EOF from 127.27.174.67:38075 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20240502 14:06:36.690716 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.665885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:38075)
0502 14:06:36.669188 (+ 3303us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:36.669264 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:36.675752 (+ 6488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:36.676849 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:36.676872 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:36.677620 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:36.678639 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.678684 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.680078 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.680100 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:36.680905 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.680940 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.681283 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.682215 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:36.682281 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:36.688307 (+ 6026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2763,"thread_start_us":2157,"threads_started":1}
W20240502 14:06:36.692081 3734 heartbeater.cc:644] Failed to heartbeat to 127.27.174.126:41625 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.174.126:41625: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
I20240502 14:06:36.694679 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.666377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.27.174.126:41625)
0502 14:06:36.668642 (+ 2265us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.668667 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.671748 (+ 3081us) server_negotiation.cc:407] Connection header received
0502 14:06:36.672216 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.672234 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.672415 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.672555 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.676331 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.677366 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.678968 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.679534 (+ 566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.682306 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:36.682398 (+ 92us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:36.682426 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:06:36.682528 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:36.688037 (+ 5509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:36.689184 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:36.689208 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:36.689251 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:36.689416 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:06:36.689970 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:06:36.689991 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:06:36.690005 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:06:36.690775 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:36.691276 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:36.691924 (+ 648us) server_negotiation.cc:299] Negotiation successful
0502 14:06:36.692501 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1591,"thread_start_us":1321,"threads_started":1}
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:36.704849 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.693597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:44101)
0502 14:06:36.694270 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:36.694316 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:36.694529 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:36.695873 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:36.695922 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:36.696454 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:36.696913 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.696957 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.698473 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.698507 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:36.699284 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.699320 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.699534 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.700247 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:36.700302 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:36.703409 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20240502 14:06:36.704924 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:36.694265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44101 (local address 127.27.174.126:41625)
0502 14:06:36.695105 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.695133 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.695180 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:06:36.695305 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.695334 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.695567 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.695792 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.697144 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.698258 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.699501 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.699913 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.703760 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:44101: BlockingRecv error: recv got EOF from 127.27.174.67:44101 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:36 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:36.715608 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:36.706115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:59807)
0502 14:06:36.706718 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:36.706761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:36.706959 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:36.707565 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:36.707590 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:36.708071 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:36.708487 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.708522 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.709785 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.709817 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:36.710710 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:36.710764 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.710969 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.711717 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:36.711785 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:36.714742 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20240502 14:06:36.716162 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:36.706260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:59807 (local address 127.27.174.126:41625)
0502 14:06:36.706898 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:36.706921 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:36.706959 (+ 38us) server_negotiation.cc:407] Connection header received
0502 14:06:36.707164 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:36.707188 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:36.707346 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:36.707483 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:36.708708 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.709582 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:36.710941 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:36.711408 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:36.715108 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:59807: BlockingRecv error: recv got EOF from 127.27.174.67:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20240502 14:06:36.716162 3734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:37.061641 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:37.050109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:56925 (local address 127.27.174.126:41625)
0502 14:06:37.050877 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:37.050907 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:37.051907 (+ 1000us) server_negotiation.cc:407] Connection header received
0502 14:06:37.052229 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:37.052255 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:37.052481 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:37.052602 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:37.054168 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.055036 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.056056 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.056382 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.060469 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:56925: BlockingRecv error: recv got EOF from 127.27.174.66:56925 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20240502 14:06:37.061661 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:37.049977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:56925)
0502 14:06:37.051764 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:37.051798 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:37.052026 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:37.052677 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:37.052694 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:37.053512 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:37.053948 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.053980 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.055199 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.055231 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:37.055881 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.055913 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.056113 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.056835 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:37.056893 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:37.060120 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1361,"thread_start_us":1081,"threads_started":1}
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:37.388850 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:37.377717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:45243 (local address 127.27.174.126:41625)
0502 14:06:37.378596 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:37.378617 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:37.379331 (+ 714us) server_negotiation.cc:407] Connection header received
0502 14:06:37.379669 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:37.379690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:37.379848 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:37.379987 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:37.381376 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.382241 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.383391 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.383882 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.387671 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:45243: BlockingRecv error: recv got EOF from 127.27.174.65:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20240502 14:06:37.388885 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:37.377384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:45243)
0502 14:06:37.379176 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:37.379213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:37.379453 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:37.380086 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:37.380110 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:37.380707 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:37.381146 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.381181 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.382439 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.382462 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:37.383177 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.383212 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.383484 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.384194 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:37.384248 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:37.387318 (+ 3070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":1121,"threads_started":1}
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:37 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:37.729071 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:37.717299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37757 (local address 127.27.174.126:41625)
0502 14:06:37.717793 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:37.717810 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:37.719265 (+ 1455us) server_negotiation.cc:407] Connection header received
0502 14:06:37.719524 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:37.719543 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:37.719707 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:37.719846 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:37.721656 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.722668 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.724031 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.724423 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.728305 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:37757: BlockingRecv error: recv got EOF from 127.27.174.67:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20240502 14:06:37.729079 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:37.717059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:37757)
0502 14:06:37.719123 (+ 2064us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:37.719162 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:37.719357 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:37.719926 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:37.719944 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:37.720726 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:37.721332 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.721383 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.722929 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:37.722969 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:37.723816 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:37.723853 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:37.724084 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:37.724847 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:37.724902 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:37.728014 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1627,"thread_start_us":1371,"threads_started":1}
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:38.074162 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:38.063144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48113 (local address 127.27.174.126:41625)
0502 14:06:38.063955 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:38.063985 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:38.064714 (+ 729us) server_negotiation.cc:407] Connection header received
0502 14:06:38.064949 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:38.064966 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:38.065150 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:38.065279 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:38.066492 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.067249 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.068342 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.068874 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.073136 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:48113: BlockingRecv error: recv got EOF from 127.27.174.66:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20240502 14:06:38.074165 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:38.063016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:48113)
0502 14:06:38.064572 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:38.064608 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:38.064799 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:38.065347 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:38.065365 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:38.065906 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:38.066305 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.066340 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.067449 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.067479 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:38.068123 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.068157 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.068413 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.069192 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:38.069250 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:38.072841 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1137,"thread_start_us":934,"threads_started":1}
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:38.402133 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:38.390606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57365 (local address 127.27.174.126:41625)
0502 14:06:38.391054 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:38.391071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:38.392575 (+ 1504us) server_negotiation.cc:407] Connection header received
0502 14:06:38.392894 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:38.392912 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:38.393071 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:38.393217 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:38.394569 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.395486 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.396621 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.396949 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.401280 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:57365: BlockingRecv error: recv got EOF from 127.27.174.65:57365 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20240502 14:06:38.402143 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:38.390491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:57365)
0502 14:06:38.392412 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:38.392448 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:38.392698 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:38.393308 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:38.393332 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:38.393916 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:38.394325 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.394363 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.395681 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.395703 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:38.396420 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.396454 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.396676 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.397362 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:38.397418 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:38.400985 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1486,"thread_start_us":1259,"threads_started":1}
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:38 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:38.741997 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:38.730646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:35529)
0502 14:06:38.732600 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:38.732638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:38.732899 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:38.733558 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:38.733588 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:38.734225 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:38.734642 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.734677 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.735888 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.735913 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:38.736618 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:38.736653 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.736927 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.737710 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:38.737765 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:38.740979 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":1262,"threads_started":1}
W20240502 14:06:38.742204 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:38.732166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35529 (local address 127.27.174.126:41625)
0502 14:06:38.732823 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:38.732848 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:38.732894 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:06:38.733099 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:38.733120 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:38.733286 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:38.733433 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:38.734885 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.735677 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:38.736847 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:38.737370 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:38.741317 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:35529: BlockingRecv error: recv got EOF from 127.27.174.67:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:39.086545 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:39.075536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:39337)
0502 14:06:39.077041 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:39.077073 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:39.077288 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:39.077872 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:39.077895 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:39.078506 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:39.078943 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.078978 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.080391 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.080415 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:39.081188 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.081224 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.081537 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.082344 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:39.082408 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:39.085488 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":889,"threads_started":1}
W20240502 14:06:39.086609 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:39.075668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:39337 (local address 127.27.174.126:41625)
0502 14:06:39.076451 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:39.076476 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:39.077207 (+ 731us) server_negotiation.cc:407] Connection header received
0502 14:06:39.077484 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:39.077501 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:39.077649 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:39.077782 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:39.079211 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.080178 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.081437 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.081981 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.085776 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:39337: BlockingRecv error: recv got EOF from 127.27.174.66:39337 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:39.414721 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:39.403555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:49531)
0502 14:06:39.405409 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:39.405447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:39.405635 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:39.406160 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:39.406179 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:39.406711 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:39.407134 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.407169 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.408325 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.408359 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:39.409187 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.409232 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.409444 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.410286 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:39.410342 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:39.413731 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1429,"thread_start_us":1189,"threads_started":1}
W20240502 14:06:39.414799 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:39.403645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:49531 (local address 127.27.174.126:41625)
0502 14:06:39.404079 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:39.404092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:39.405534 (+ 1442us) server_negotiation.cc:407] Connection header received
0502 14:06:39.405783 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:39.405799 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:39.405952 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:39.406126 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:39.407337 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.408147 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.409434 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.409933 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.414016 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:49531: BlockingRecv error: recv got EOF from 127.27.174.65:49531 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20240502 14:06:39.615406 3305 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac620 after lost signal to thread 3241
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:39 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:39.755940 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:39.743741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34819 (local address 127.27.174.126:41625)
0502 14:06:39.744433 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:39.744460 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:39.745987 (+ 1527us) server_negotiation.cc:407] Connection header received
0502 14:06:39.746336 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:39.746355 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:39.746511 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:39.746649 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:39.747967 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.748800 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.749976 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.750519 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.755024 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:34819: BlockingRecv error: recv got EOF from 127.27.174.67:34819 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20240502 14:06:39.756464 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:39.743567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:34819)
0502 14:06:39.745849 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:39.745886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:39.746110 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:39.746746 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:39.746771 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:39.747341 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:39.747733 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.747770 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.749008 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:39.749032 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:39.749740 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:39.749777 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:39.750082 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:39.750915 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:39.750970 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:39.754796 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1848,"thread_start_us":1597,"threads_started":1}
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:40.100875 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:40.088171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:56447)
0502 14:06:40.089820 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:40.089856 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:40.090094 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:40.090886 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:40.090903 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:40.091411 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:40.091778 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.091813 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.094189 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.094230 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:40.095072 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.095249 (+ 177us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.095540 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.096386 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:40.096440 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:40.099916 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1206,"thread_start_us":982,"threads_started":1}
W20240502 14:06:40.102283 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:40.088737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:56447 (local address 127.27.174.126:41625)
0502 14:06:40.089822 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:40.089858 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:40.090015 (+ 157us) server_negotiation.cc:407] Connection header received
0502 14:06:40.090371 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:40.090398 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:40.090680 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:40.090869 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:40.092085 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.093895 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.095484 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.096882 (+ 1398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.100411 (+ 3529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:56447: BlockingRecv error: recv got EOF from 127.27.174.66:56447 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:40.430759 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:40.416349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:52339 (local address 127.27.174.126:41625)
0502 14:06:40.417221 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:40.417257 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:40.418678 (+ 1421us) server_negotiation.cc:407] Connection header received
0502 14:06:40.419067 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:40.419088 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:40.419252 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:40.419388 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:40.421361 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.422269 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.424111 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.424469 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.429451 (+ 4982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:52339: BlockingRecv error: recv got EOF from 127.27.174.65:52339 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:06:40.430758 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:40.416242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:52339)
0502 14:06:40.418499 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:40.418557 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:40.418870 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:40.419527 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:40.419555 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:40.420414 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:40.421101 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.421160 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.422489 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.422530 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:40.423771 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.423837 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.424156 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.425008 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:40.425096 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:40.429071 (+ 3975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1508,"thread_start_us":1113,"threads_started":1}
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:40 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:40.771613 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:40.758432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:42431 (local address 127.27.174.126:41625)
0502 14:06:40.759326 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:40.759363 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:40.760543 (+ 1180us) server_negotiation.cc:407] Connection header received
0502 14:06:40.760874 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:40.760893 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:40.761048 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:40.761172 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:40.763131 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.763902 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.765354 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.765870 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.770575 (+ 4705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:42431: BlockingRecv error: recv got EOF from 127.27.174.67:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20240502 14:06:40.771662 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:40.758297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:42431)
0502 14:06:40.760370 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:40.760430 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:40.760690 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:40.761285 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:40.761318 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:40.762227 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:40.762860 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.762928 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.764103 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:40.764125 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:40.765050 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:40.765098 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:40.765436 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:40.766347 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:40.766442 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:40.770249 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":1100,"threads_started":1}
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:41.115216 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:41.102472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:35113 (local address 127.27.174.126:41625)
0502 14:06:41.103360 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:41.103391 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:41.104666 (+ 1275us) server_negotiation.cc:407] Connection header received
0502 14:06:41.104936 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:41.104975 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:41.105207 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:41.105387 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:41.106669 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.108054 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.109401 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.110014 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.114134 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:35113: BlockingRecv error: recv got EOF from 127.27.174.66:35113 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20240502 14:06:41.116079 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:41.102355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:35113)
0502 14:06:41.104496 (+ 2141us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:41.104532 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:41.104753 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:41.105437 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:41.105455 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:41.105984 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:41.106443 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.106478 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.108251 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.108274 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:41.109171 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.109209 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.109432 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.110197 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:41.110261 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:41.113829 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":1383,"threads_started":1}
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:41.449522 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:41.432975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47367 (local address 127.27.174.126:41625)
0502 14:06:41.433938 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:41.433974 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:41.435656 (+ 1682us) server_negotiation.cc:407] Connection header received
0502 14:06:41.435956 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:41.435974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:41.436132 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:41.436276 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:41.437605 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.438441 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.439624 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.440170 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.448271 (+ 8101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:47367: BlockingRecv error: recv got EOF from 127.27.174.65:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20240502 14:06:41.449544 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:41.432706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:47367)
0502 14:06:41.435499 (+ 2793us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:41.435537 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:41.435765 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:41.436387 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:41.436412 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:41.436983 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:41.437368 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.437405 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.438646 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.438669 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:41.439392 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.439428 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.439699 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.444742 (+ 5043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:41.444800 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:41.447898 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2238,"thread_start_us":2027,"threads_started":1}
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:41 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:41.785029 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:41.773421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:41287 (local address 127.27.174.126:41625)
0502 14:06:41.773886 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:41.773903 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:41.775523 (+ 1620us) server_negotiation.cc:407] Connection header received
0502 14:06:41.775781 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:41.775798 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:41.775955 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:41.776093 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:41.777319 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.778146 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.779553 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.780062 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.783909 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:41287: BlockingRecv error: recv got EOF from 127.27.174.67:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20240502 14:06:41.785073 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:41.773278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:41287)
0502 14:06:41.775380 (+ 2102us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:41.775418 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:41.775622 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:41.776176 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:41.776194 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:41.776731 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:41.777116 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.777154 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.778391 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:41.778429 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:41.779284 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:41.779336 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:41.779626 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:41.780442 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:41.780506 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:41.783579 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":1377,"threads_started":1}
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:42.129406 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:42.117797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:37325)
0502 14:06:42.119961 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:42.119999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:42.120235 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:42.120863 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:42.120882 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:42.121429 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:42.121989 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.122073 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.123512 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.123535 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:42.124259 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.124292 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.124495 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.125178 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:42.125243 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:42.128411 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":1426,"threads_started":1}
W20240502 14:06:42.129472 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:42.117973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:37325 (local address 127.27.174.126:41625)
0502 14:06:42.118641 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:42.118667 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:42.120111 (+ 1444us) server_negotiation.cc:407] Connection header received
0502 14:06:42.120438 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:42.120464 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:42.120664 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:42.120785 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:42.122345 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.123343 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.124434 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.124757 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.128690 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:37325: BlockingRecv error: recv got EOF from 127.27.174.66:37325 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:42.464701 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:42.451612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:32803 (local address 127.27.174.126:41625)
0502 14:06:42.452317 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:42.452349 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:42.453533 (+ 1184us) server_negotiation.cc:407] Connection header received
0502 14:06:42.453867 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:42.453886 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:42.454106 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:42.454247 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:42.456094 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.456893 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.458065 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.458391 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.463543 (+ 5152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:32803: BlockingRecv error: recv got EOF from 127.27.174.65:32803 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20240502 14:06:42.464901 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:42.451592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:32803)
0502 14:06:42.453344 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:42.453414 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:42.453713 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:42.454360 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:42.454383 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:42.455100 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:42.455790 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.455843 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.457056 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.457076 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:42.457872 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.457913 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.458180 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.459284 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:42.459348 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:42.463200 (+ 3852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":925,"threads_started":1}
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:42 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:42.797802 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:42.786713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40163 (local address 127.27.174.126:41625)
0502 14:06:42.787535 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:42.787564 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:42.788354 (+ 790us) server_negotiation.cc:407] Connection header received
0502 14:06:42.788633 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:42.788649 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:42.788798 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:42.788923 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:42.790381 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.791162 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.792291 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.792783 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.796651 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:40163: BlockingRecv error: recv got EOF from 127.27.174.67:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:06:42.797832 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:42.786465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:40163)
0502 14:06:42.788201 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:42.788244 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:42.788457 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:42.789059 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:42.789085 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:42.789672 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:42.790123 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.790177 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.791369 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:42.791391 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:42.792077 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:42.792109 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:42.792371 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:42.793090 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:42.793143 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:42.796288 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":969,"threads_started":1}
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:43.141842 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:43.130834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:49345)
0502 14:06:43.132282 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:43.132313 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:43.132521 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:43.133073 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:43.133095 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:43.133696 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:43.134180 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.134269 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.135700 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.135722 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:43.136412 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.136444 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.136668 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.137342 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:43.137400 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:43.140466 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":857,"threads_started":1}
W20240502 14:06:43.142325 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:43.130974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:49345 (local address 127.27.174.126:41625)
0502 14:06:43.131736 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:43.131755 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:43.132416 (+ 661us) server_negotiation.cc:407] Connection header received
0502 14:06:43.132687 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:43.132704 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:43.132843 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:43.132982 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:43.134472 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.135516 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.136597 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.136922 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.140873 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:49345: BlockingRecv error: recv got EOF from 127.27.174.66:49345 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:43.477700 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:43.466689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42587 (local address 127.27.174.126:41625)
0502 14:06:43.467185 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:43.467202 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:43.467938 (+ 736us) server_negotiation.cc:407] Connection header received
0502 14:06:43.468185 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:43.468204 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:43.468365 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:43.468503 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:43.469947 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.470927 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.472193 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.472712 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.476529 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:42587: BlockingRecv error: recv got EOF from 127.27.174.65:42587 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
I20240502 14:06:43.477718 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:43.466551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:42587)
0502 14:06:43.467815 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:43.467850 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:43.468036 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:43.468576 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:43.468591 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:43.469185 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:43.469683 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.469739 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.471126 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.471148 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:43.471979 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.472014 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.472280 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.473101 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:43.473164 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:43.476153 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":578,"threads_started":1}
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:43 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:43.810739 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:43.799324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37783 (local address 127.27.174.126:41625)
0502 14:06:43.799947 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:43.799966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:43.801376 (+ 1410us) server_negotiation.cc:407] Connection header received
0502 14:06:43.801633 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:43.801652 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:43.801852 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:43.801967 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:43.803229 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.803938 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.805241 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.805539 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.809650 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:37783: BlockingRecv error: recv got EOF from 127.27.174.67:37783 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20240502 14:06:43.810869 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:43.799248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:37783)
0502 14:06:43.801253 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:43.801290 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:43.801484 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:43.802056 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:43.802075 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:43.802637 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:43.803044 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.803079 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.804093 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:43.804114 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:43.804999 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:43.805050 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:43.805331 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:43.806020 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:43.806115 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:43.809297 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1354,"threads_started":1}
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:44.156203 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:44.143732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:34129 (local address 127.27.174.126:41625)
0502 14:06:44.144409 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:44.144433 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:44.145638 (+ 1205us) server_negotiation.cc:407] Connection header received
0502 14:06:44.146056 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:44.146073 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:44.146227 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:44.146352 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:44.148226 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.149017 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.150386 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.150930 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.155576 (+ 4646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:34129: BlockingRecv error: recv got EOF from 127.27.174.66:34129 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20240502 14:06:44.156458 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:44.143665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:34129)
0502 14:06:44.145453 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:44.145495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:44.145842 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:44.146476 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:44.146510 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:44.147286 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:44.147973 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.148021 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.149242 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.149265 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:44.150130 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.150176 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.150502 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.151325 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:44.151411 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:44.155328 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":1014,"threads_started":1}
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:44.489795 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:44.479636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42707 (local address 127.27.174.126:41625)
0502 14:06:44.480194 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:44.480226 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:44.481019 (+ 793us) server_negotiation.cc:407] Connection header received
0502 14:06:44.481341 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:44.481364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:44.481558 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:44.481678 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:44.483162 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.483933 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.485040 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.485355 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.489159 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:42707: BlockingRecv error: recv got EOF from 127.27.174.65:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20240502 14:06:44.489897 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:44.479448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:42707)
0502 14:06:44.480867 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:44.480913 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:44.481154 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:44.481801 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:44.481828 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:44.482503 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:44.482933 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.482980 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.484092 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.484122 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:44.484870 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.484904 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.485113 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.485831 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:44.485884 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:44.488869 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":698,"threads_started":1}
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:44 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:44.823647 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:44.812454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:39071 (local address 127.27.174.126:41625)
0502 14:06:44.812817 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:44.812830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:44.814495 (+ 1665us) server_negotiation.cc:407] Connection header received
0502 14:06:44.814759 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:44.814782 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:44.814975 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:44.815130 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:44.816335 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.817106 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.818180 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.818499 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.822600 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:39071: BlockingRecv error: recv got EOF from 127.27.174.67:39071 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20240502 14:06:44.823836 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:44.812346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:39071)
0502 14:06:44.814369 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:44.814407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:44.814600 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:44.815197 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:44.815214 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:44.815752 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:44.816133 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.816170 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.817261 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:44.817283 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:44.817967 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:44.818000 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:44.818250 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:44.818956 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:44.819010 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:44.822268 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1548,"thread_start_us":1308,"threads_started":1}
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:45.168537 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:45.158019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:41289 (local address 127.27.174.126:41625)
0502 14:06:45.158616 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:45.158635 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:45.159018 (+ 383us) server_negotiation.cc:407] Connection header received
0502 14:06:45.159264 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:45.159278 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:45.159417 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:45.159551 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:45.160837 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.161600 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.162787 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.163163 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.167532 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:41289: BlockingRecv error: recv got EOF from 127.27.174.66:41289 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20240502 14:06:45.168710 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:45.157909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:41289)
0502 14:06:45.158899 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:45.158932 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:45.159107 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:45.159615 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:45.159634 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:45.160180 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:45.160641 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.160679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.161745 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.161774 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:45.162559 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.162595 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.162863 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.163691 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:45.163749 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:45.167202 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":410,"threads_started":1}
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:45.501734 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:45.491755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42045 (local address 127.27.174.126:41625)
0502 14:06:45.492302 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:45.492323 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:45.492972 (+ 649us) server_negotiation.cc:407] Connection header received
0502 14:06:45.493256 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:45.493271 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:45.493392 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:45.493505 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:45.494922 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.495699 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.496794 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.497280 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.501155 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:42045: BlockingRecv error: recv got EOF from 127.27.174.65:42045 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20240502 14:06:45.501835 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:45.491542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:42045)
0502 14:06:45.492837 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:45.492872 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:45.493080 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:45.493600 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:45.493623 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:45.494197 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:45.494687 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.494739 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.495904 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.495927 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:45.496596 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.496627 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.496872 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.497648 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:45.497700 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:45.500919 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":565,"threads_started":1}
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:45 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:45.837594 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:45.825593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34437 (local address 127.27.174.126:41625)
0502 14:06:45.826058 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:45.826079 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:45.827773 (+ 1694us) server_negotiation.cc:407] Connection header received
0502 14:06:45.828083 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:45.828103 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:45.828250 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:45.828354 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:45.830031 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.831037 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.832450 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.832801 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.836965 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:34437: BlockingRecv error: recv got EOF from 127.27.174.67:34437 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20240502 14:06:45.837812 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:45.825539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:34437)
0502 14:06:45.827635 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:45.827675 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:45.827929 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:45.828426 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:45.828449 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:45.829104 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:45.829827 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.829874 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.831255 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:45.831278 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:45.832225 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:45.832275 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:45.832519 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:45.833348 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:45.833408 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:45.836739 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1614,"thread_start_us":1284,"threads_started":1}
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:46.179961 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:46.170349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:46905 (local address 127.27.174.126:41625)
0502 14:06:46.170937 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:46.170958 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:46.171374 (+ 416us) server_negotiation.cc:407] Connection header received
0502 14:06:46.171629 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:46.171642 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:46.171764 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:46.171868 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:46.173296 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.174255 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.175373 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.175697 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.179336 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:46905: BlockingRecv error: recv got EOF from 127.27.174.66:46905 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:06:46.180064 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:46.170244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:46905)
0502 14:06:46.171246 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:46.171281 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:46.171490 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:46.171985 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:46.172012 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:46.172680 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:46.173059 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.173094 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.174451 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.174489 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:46.175189 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.175224 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.175434 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.176170 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:46.176222 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:46.179079 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":380,"threads_started":1}
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:46.518162 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:46.503352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:45891 (local address 127.27.174.126:41625)
0502 14:06:46.503705 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:46.503719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:46.504639 (+ 920us) server_negotiation.cc:407] Connection header received
0502 14:06:46.504867 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:46.504880 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:46.505000 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:46.505123 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:46.506352 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.507104 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.508297 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.508607 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.517503 (+ 8896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:45891: BlockingRecv error: recv got EOF from 127.27.174.65:45891 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20240502 14:06:46.518224 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:46.503279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:45891)
0502 14:06:46.504522 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:46.504556 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:46.504732 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:46.505188 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:46.505205 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:46.505726 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:46.506160 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.506202 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.507257 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.507296 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:46.508103 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.508138 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.508344 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.514102 (+ 5758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:46.514165 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:46.517248 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":537,"threads_started":1}
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:46 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:46.850384 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:46.839581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37093 (local address 127.27.174.126:41625)
0502 14:06:46.839948 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:46.839963 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:46.840960 (+ 997us) server_negotiation.cc:407] Connection header received
0502 14:06:46.841316 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:46.841341 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:46.841494 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:46.841626 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:46.843020 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.843820 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.844986 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.845507 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.849296 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:37093: BlockingRecv error: recv got EOF from 127.27.174.67:37093 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20240502 14:06:46.850593 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:46.839450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:37093)
0502 14:06:46.840811 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:46.840859 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:46.841104 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:46.841723 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:46.841748 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:46.842367 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:46.842774 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.842808 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.844024 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:46.844048 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:46.844771 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:46.844804 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:46.845064 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:46.845834 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:46.845890 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:46.848958 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":606,"threads_started":1}
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:47.191896 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:47.181874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58627 (local address 127.27.174.126:41625)
0502 14:06:47.182646 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:47.182667 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:47.183312 (+ 645us) server_negotiation.cc:407] Connection header received
0502 14:06:47.183602 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:47.183618 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:47.183750 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:47.183863 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:47.185236 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.186099 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.187165 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.187451 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.191278 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:58627: BlockingRecv error: recv got EOF from 127.27.174.66:58627 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20240502 14:06:47.191985 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:47.181814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:58627)
0502 14:06:47.183174 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:47.183209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:47.183418 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:47.183980 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:47.184017 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:47.184636 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:47.185032 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.185067 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.186278 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.186299 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:47.187004 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.187035 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.187245 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.187907 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:47.187980 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:47.191030 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":630,"threads_started":1}
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:47.531119 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:47.520477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:35373 (local address 127.27.174.126:41625)
0502 14:06:47.520972 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:47.520997 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:47.521683 (+ 686us) server_negotiation.cc:407] Connection header received
0502 14:06:47.522099 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:47.522122 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:47.522310 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:47.522433 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:47.524137 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.524934 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.526030 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.526379 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.530509 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:35373: BlockingRecv error: recv got EOF from 127.27.174.65:35373 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20240502 14:06:47.531255 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:47.520370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:35373)
0502 14:06:47.521536 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:47.521570 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:47.521843 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:47.522554 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:47.522586 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:47.523370 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:47.523868 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.523922 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.525097 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.525119 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:47.525847 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.525881 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.526130 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.526847 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:47.526903 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:47.530249 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":496,"threads_started":1}
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:47 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:47.862705 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:47.852063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:50999 (local address 127.27.174.126:41625)
0502 14:06:47.852577 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:47.852597 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:47.853348 (+ 751us) server_negotiation.cc:407] Connection header received
0502 14:06:47.853679 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:47.853694 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:47.853823 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:47.853946 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:47.855285 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.856040 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.857129 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.857571 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.861829 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:50999: BlockingRecv error: recv got EOF from 127.27.174.67:50999 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20240502 14:06:47.862710 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:47.851960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:50999)
0502 14:06:47.853210 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:47.853246 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:47.853488 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:47.854071 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:47.854095 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:47.854690 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:47.855068 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.855102 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.856228 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:47.856251 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:47.856910 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:47.856948 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:47.857204 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:47.858020 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:47.858110 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:47.861456 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":477,"threads_started":1}
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:48.204392 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:48.193788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:54617 (local address 127.27.174.126:41625)
0502 14:06:48.194371 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:48.194391 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:48.194879 (+ 488us) server_negotiation.cc:407] Connection header received
0502 14:06:48.195245 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:48.195260 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:48.195384 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:48.195503 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:48.196843 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.197604 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.198756 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.199133 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.203454 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:54617: BlockingRecv error: recv got EOF from 127.27.174.66:54617 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:06:48.204571 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:48.193498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:54617)
0502 14:06:48.194753 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:48.194787 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:48.195009 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:48.195606 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:48.195624 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:48.196211 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:48.196641 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.196682 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.197758 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.197776 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:48.198571 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.198605 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.198859 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.199702 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:48.199761 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:48.203124 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":525,"threads_started":1}
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:48.542857 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:48.532888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:54595 (local address 127.27.174.126:41625)
0502 14:06:48.533490 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:48.533511 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:48.533933 (+ 422us) server_negotiation.cc:407] Connection header received
0502 14:06:48.534185 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:48.534199 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:48.534330 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:48.534441 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:48.535652 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.536413 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.537468 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.537760 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.542219 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:54595: BlockingRecv error: recv got EOF from 127.27.174.65:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20240502 14:06:48.542955 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:48.532825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:54595)
0502 14:06:48.533802 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:48.533836 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:48.534021 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:48.534521 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:48.534540 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:48.535070 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:48.535456 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.535501 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.536574 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.536595 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:48.537288 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.537321 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.537542 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.538326 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:48.538393 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:48.541927 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":374,"threads_started":1}
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:48 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:48.874946 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:48.864352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35375 (local address 127.27.174.126:41625)
0502 14:06:48.864900 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:48.864924 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:48.865548 (+ 624us) server_negotiation.cc:407] Connection header received
0502 14:06:48.865880 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:48.865897 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:48.866087 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:48.866266 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:48.868031 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.869039 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.870340 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.870672 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.874313 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:35375: BlockingRecv error: recv got EOF from 127.27.174.67:35375 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20240502 14:06:48.874971 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:48.864238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:35375)
0502 14:06:48.865404 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:48.865436 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:48.865656 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:48.866365 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:48.866397 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:48.867231 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:48.867781 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.867836 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.869237 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:48.869261 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:48.870146 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:48.870184 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:48.870399 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:48.871083 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:48.871137 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:48.874019 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":504,"threads_started":1}
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:49.216585 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:49.206485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57473 (local address 127.27.174.126:41625)
0502 14:06:49.207039 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:49.207062 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:49.207869 (+ 807us) server_negotiation.cc:407] Connection header received
0502 14:06:49.208230 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:49.208247 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:49.208378 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:49.208520 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:49.210073 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.210878 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.211880 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.212159 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.215979 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:57473: BlockingRecv error: recv got EOF from 127.27.174.66:57473 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20240502 14:06:49.216689 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:49.206328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:57473)
0502 14:06:49.207713 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:49.207768 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:49.207999 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:49.208630 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:49.208648 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:49.209351 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:49.209841 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.209883 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.211020 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.211040 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:49.211718 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.211749 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.211947 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.212610 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:49.212663 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:49.215736 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":548,"threads_started":1}
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:49.554832 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:49.544626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:41275 (local address 127.27.174.126:41625)
0502 14:06:49.544979 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:49.544992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:49.545886 (+ 894us) server_negotiation.cc:407] Connection header received
0502 14:06:49.546165 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:49.546179 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:49.546321 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:49.546437 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:49.547700 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.548575 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.549959 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.550356 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.554202 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:41275: BlockingRecv error: recv got EOF from 127.27.174.65:41275 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20240502 14:06:49.554930 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:49.544510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:41275)
0502 14:06:49.545752 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:49.545788 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:49.545977 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:49.546520 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:49.546539 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:49.547073 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:49.547476 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.547510 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.548775 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.548799 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:49.549758 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.549795 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.550010 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.550779 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:49.550837 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:49.553912 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":580,"threads_started":1}
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:49 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:49.885896 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:49.876328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:47027 (local address 127.27.174.126:41625)
0502 14:06:49.876988 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:49.877009 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:49.877606 (+ 597us) server_negotiation.cc:407] Connection header received
0502 14:06:49.877829 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:49.877841 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:49.877963 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:49.878097 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:49.879254 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.880022 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.881079 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.881387 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.885302 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:47027: BlockingRecv error: recv got EOF from 127.27.174.67:47027 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20240502 14:06:49.886056 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:49.876255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:47027)
0502 14:06:49.877489 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:49.877521 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:49.877695 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:49.878159 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:49.878176 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:49.878711 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:49.879073 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.879110 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.880175 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:49.880198 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:49.880924 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:49.880954 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:49.881143 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:49.881872 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:49.881947 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:49.885012 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":655,"threads_started":1}
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
I20240502 14:06:50.229705 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:50.218258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:58437)
0502 14:06:50.219502 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:50.219543 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:50.219805 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:50.220498 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:50.220532 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:50.221165 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:50.221542 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.221592 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.223601 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.223629 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:50.224555 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.224593 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.224823 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.225553 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:50.225610 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:50.228721 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":382,"threads_started":1}
W20240502 14:06:50.229805 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:50.218633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58437 (local address 127.27.174.126:41625)
0502 14:06:50.219435 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:50.219462 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:50.219655 (+ 193us) server_negotiation.cc:407] Connection header received
0502 14:06:50.220056 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:50.220072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:50.220263 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:50.220421 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:50.221842 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.223362 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.224762 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.225089 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.229005 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:58437: BlockingRecv error: recv got EOF from 127.27.174.66:58437 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:50.567080 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:50.556383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:42575 (local address 127.27.174.126:41625)
0502 14:06:50.556930 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:50.556952 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:50.557544 (+ 592us) server_negotiation.cc:407] Connection header received
0502 14:06:50.557847 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:50.557862 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:50.557992 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:50.558132 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:50.559404 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.560192 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.561314 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.561818 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.566121 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:42575: BlockingRecv error: recv got EOF from 127.27.174.65:42575 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20240502 14:06:50.567248 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:50.556303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:42575)
0502 14:06:50.557402 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:50.557436 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:50.557667 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:50.558234 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:50.558260 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:50.558818 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:50.559183 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.559218 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.560389 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.560412 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:50.561115 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.561145 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.561411 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.562317 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:50.562392 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:50.565773 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":466,"threads_started":1}
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:50 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.67@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:50.898248 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:50.888160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:45279 (local address 127.27.174.126:41625)
0502 14:06:50.888738 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:50.888753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:50.889031 (+ 278us) server_negotiation.cc:407] Connection header received
0502 14:06:50.889279 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:50.889293 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:50.889422 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:50.889540 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:50.890871 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.891655 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.892862 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.893289 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.897594 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.67:45279: BlockingRecv error: recv got EOF from 127.27.174.67:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20240502 14:06:50.898355 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:50.887839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.67:45279)
0502 14:06:50.888912 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:50.888947 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:50.889133 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:50.889603 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:50.889620 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:50.890253 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:50.890676 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.890715 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.891809 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:50.891830 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:50.892669 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:50.892704 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:50.892936 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:50.893761 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:50.893818 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:50.897335 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":454,"threads_started":1}
May 02 14:06:51 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:51 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.66@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:51.241309 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:51.231319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:46945 (local address 127.27.174.126:41625)
0502 14:06:51.231853 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:51.231875 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:51.232482 (+ 607us) server_negotiation.cc:407] Connection header received
0502 14:06:51.232757 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:51.232772 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:51.232903 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:51.233031 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:51.234362 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.235113 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:51.236235 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.236734 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:51.240698 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.66:46945: BlockingRecv error: recv got EOF from 127.27.174.66:46945 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20240502 14:06:51.241413 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:51.231172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.66:46945)
0502 14:06:51.232335 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:51.232369 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:51.232583 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:51.233120 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:51.233142 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:51.233711 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:51.234142 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:51.234178 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.235296 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:51.235318 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:51.236014 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:51.236046 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.236331 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:51.237174 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:51.237240 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:51.240446 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":498,"threads_started":1}
May 02 14:06:51 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
May 02 14:06:51 dist-test-slave-dlb8 krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.174.65@KRBTEST.COM for oryx/127.27.174.126@KRBTEST.COM, Server not found in Kerberos database
W20240502 14:06:51.582830 3614 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:51.568714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:47183 (local address 127.27.174.126:41625)
0502 14:06:51.569304 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:51.569326 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:51.570030 (+ 704us) server_negotiation.cc:407] Connection header received
0502 14:06:51.570346 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:51.570361 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:51.570501 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:51.570615 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:51.571811 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.572574 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:51.573681 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.574005 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:51.582197 (+ 8192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.174.65:47183: BlockingRecv error: recv got EOF from 127.27.174.65:47183 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20240502 14:06:51.582940 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:51.568651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41625 (local address 127.27.174.65:47183)
0502 14:06:51.569871 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:51.569922 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:51.570181 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:51.570682 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:51.570701 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:51.571246 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:51.571619 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:51.571655 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.572740 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:51.572762 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:51.573491 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:51.573545 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:51.573753 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:51.578680 (+ 4927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:06:51.578734 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:51.581883 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.174.126:41625: Server oryx/127.27.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":376,"threads_started":1}
I20240502 14:06:51.675879 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3311
I20240502 14:06:51.699286 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3464
I20240502 14:06:51.723102 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3600
I20240502 14:06:51.747081 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3240
2024-05-02T14:06:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (30762 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-02T14:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:06:51Z Disabled control of system clock
I20240502 14:06:51.837029 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44691
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240502 14:06:52.120970 3794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:52.121555 3794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:52.122093 3794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:52.151764 3794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:52.152089 3794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:52.152290 3794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:52.152513 3794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:52.152727 3794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:52.185475 3794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:44691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:52.186648 3794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:52.188177 3794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:52.204456 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:52.205099 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:52.205653 3794 server_base.cc:1034] running on GCE node
W20240502 14:06:52.205204 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:53.273981 3794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:53.276625 3794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:53.277957 3794 hybrid_clock.cc:648] HybridClock initialized: now 1714658813277932 us; error 43 us; skew 500 ppm
I20240502 14:06:53.278820 3794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:53.284902 3794 webserver.cc:460] Webserver started at http://127.27.174.126:38711/ using document root <none> and password file <none>
I20240502 14:06:53.285784 3794 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:53.285987 3794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:53.286487 3794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:53.291457 3794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "cda3ce0dafca49bb910f09f7252b05b4"
format_stamp: "Formatted at 2024-05-02 14:06:53 on dist-test-slave-dlb8"
server_key: "62687a3ffa6c52e2ca86a55c8b2ed23f"
server_key_iv: "659575dc832ccee722a4146063b553d1"
server_key_version: "encryptionkey@0"
I20240502 14:06:53.292551 3794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "cda3ce0dafca49bb910f09f7252b05b4"
format_stamp: "Formatted at 2024-05-02 14:06:53 on dist-test-slave-dlb8"
server_key: "62687a3ffa6c52e2ca86a55c8b2ed23f"
server_key_iv: "659575dc832ccee722a4146063b553d1"
server_key_version: "encryptionkey@0"
I20240502 14:06:53.299077 3794 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:06:53.304309 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:53.305238 3794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:06:53.305532 3794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "cda3ce0dafca49bb910f09f7252b05b4"
format_stamp: "Formatted at 2024-05-02 14:06:53 on dist-test-slave-dlb8"
server_key: "62687a3ffa6c52e2ca86a55c8b2ed23f"
server_key_iv: "659575dc832ccee722a4146063b553d1"
server_key_version: "encryptionkey@0"
I20240502 14:06:53.305864 3794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:53.376143 3794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:53.377585 3794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:53.378021 3794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:53.444948 3794 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:44691
I20240502 14:06:53.445027 3861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:44691 every 8 connection(s)
I20240502 14:06:53.447508 3794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:06:53.449503 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3794
I20240502 14:06:53.450130 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:06:53.451653 28345 external_mini_cluster.cc:1428] Setting key 48425015d04678c8e0ac8f76a104f815
I20240502 14:06:53.453104 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:53.472985 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:53.454072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.27.174.126:44691)
0502 14:06:53.456981 (+ 2909us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:53.457003 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:53.457051 (+ 48us) server_negotiation.cc:407] Connection header received
0502 14:06:53.462406 (+ 5355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:53.462442 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:53.462794 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:53.463114 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:53.464843 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:53.466251 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:53.468117 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:53.468544 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:53.469593 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:53.469680 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:53.469709 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:06:53.470531 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:53.470622 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:06:53.470666 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:53.470786 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:53.471126 (+ 340us) server_negotiation.cc:299] Negotiation successful
0502 14:06:53.471369 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2317,"thread_start_us":1909,"threads_started":1}
I20240502 14:06:53.477299 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4: Bootstrap starting.
I20240502 14:06:53.482439 3862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:53.483999 3862 log.cc:824] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:06:53.488487 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4: No bootstrap required, opened a new log
I20240502 14:06:53.507525 3862 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } }
I20240502 14:06:53.508179 3862 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:53.508440 3862 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda3ce0dafca49bb910f09f7252b05b4, State: Initialized, Role: FOLLOWER
I20240502 14:06:53.509104 3862 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } }
I20240502 14:06:53.509552 3862 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:53.509804 3862 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:53.510121 3862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:53.514421 3862 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } }
I20240502 14:06:53.515079 3862 leader_election.cc:302] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda3ce0dafca49bb910f09f7252b05b4; no voters:
I20240502 14:06:53.516649 3862 leader_election.cc:288] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:53.517402 3867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:53.519433 3867 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [term 1 LEADER]: Becoming Leader. State: Replica: cda3ce0dafca49bb910f09f7252b05b4, State: Running, Role: LEADER
I20240502 14:06:53.520063 3867 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } }
I20240502 14:06:53.520381 3862 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:53.534070 3868 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda3ce0dafca49bb910f09f7252b05b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } } }
I20240502 14:06:53.534776 3868 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:53.534929 3869 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda3ce0dafca49bb910f09f7252b05b4. Latest consensus state: current_term: 1 leader_uuid: "cda3ce0dafca49bb910f09f7252b05b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda3ce0dafca49bb910f09f7252b05b4" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 44691 } } }
I20240502 14:06:53.535545 3869 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:53.538200 3877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:53.548918 3877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:53.563556 3877 catalog_manager.cc:1347] Generated new cluster ID: d3a2cb6f61994a27a6f361f669a64f33
I20240502 14:06:53.563815 3877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:53.576148 3877 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:53.577461 3877 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:53.590189 3877 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cda3ce0dafca49bb910f09f7252b05b4: Generated new TSK 0
I20240502 14:06:53.591001 3877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:53.602780 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--builtin_ntp_servers=127.27.174.84:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240502 14:06:53.900216 3886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:53.900743 3886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:53.901295 3886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:53.929690 3886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:53.929957 3886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:53.930795 3886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:06:53.963986 3886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:53.965119 3886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:53.966662 3886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:53.983947 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:53.984352 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:53.986549 3886 server_base.cc:1034] running on GCE node
W20240502 14:06:53.986207 3895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:55.041553 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:55.044353 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:55.045763 3886 hybrid_clock.cc:648] HybridClock initialized: now 1714658815045708 us; error 73 us; skew 500 ppm
I20240502 14:06:55.046584 3886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:55.053488 3886 webserver.cc:460] Webserver started at http://127.27.174.65:39803/ using document root <none> and password file <none>
I20240502 14:06:55.054399 3886 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:55.054589 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:55.055043 3886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:55.060451 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "89f033f1cb34432f8a1917cc651e4cb0"
format_stamp: "Formatted at 2024-05-02 14:06:55 on dist-test-slave-dlb8"
server_key: "103f14cc799ee56634585e7ed39be5da"
server_key_iv: "8f181b838ca753baaaeb9502e2d9d19d"
server_key_version: "encryptionkey@0"
I20240502 14:06:55.061586 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "89f033f1cb34432f8a1917cc651e4cb0"
format_stamp: "Formatted at 2024-05-02 14:06:55 on dist-test-slave-dlb8"
server_key: "103f14cc799ee56634585e7ed39be5da"
server_key_iv: "8f181b838ca753baaaeb9502e2d9d19d"
server_key_version: "encryptionkey@0"
I20240502 14:06:55.068917 3886 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:55.074731 3902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:55.075721 3886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:55.075990 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "89f033f1cb34432f8a1917cc651e4cb0"
format_stamp: "Formatted at 2024-05-02 14:06:55 on dist-test-slave-dlb8"
server_key: "103f14cc799ee56634585e7ed39be5da"
server_key_iv: "8f181b838ca753baaaeb9502e2d9d19d"
server_key_version: "encryptionkey@0"
I20240502 14:06:55.076308 3886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:55.128789 3886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:55.130198 3886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:55.130632 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:55.133508 3886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:55.138533 3886 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:55.138845 3886 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:06:55.139212 3886 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:55.139516 3886 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:55.277702 3886 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:43413
I20240502 14:06:55.277791 4014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:43413 every 8 connection(s)
I20240502 14:06:55.280102 3886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:06:55.286399 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 3886
I20240502 14:06:55.286934 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:06:55.288470 28345 external_mini_cluster.cc:1428] Setting key 3a153ee653b4cf4c1e727454f9b1cff0
I20240502 14:06:55.294412 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--builtin_ntp_servers=127.27.174.84:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240502 14:06:55.303025 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:55.283980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39527 (local address 127.27.174.126:44691)
0502 14:06:55.285472 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:55.285507 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:55.287259 (+ 1752us) server_negotiation.cc:407] Connection header received
0502 14:06:55.293459 (+ 6200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:55.293478 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:55.293631 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:55.293758 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:55.295949 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:55.296772 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:55.298669 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:55.299009 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:55.300334 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:55.300404 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:55.300421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:06:55.300464 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:55.300534 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:06:55.300573 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:55.300684 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:55.301057 (+ 373us) server_negotiation.cc:299] Negotiation successful
0502 14:06:55.301205 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":480,"threads_started":1}
I20240502 14:06:55.303171 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:55.284172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44691 (local address 127.27.174.65:39527)
0502 14:06:55.286994 (+ 2822us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:55.287080 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:55.293104 (+ 6024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:55.294083 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:55.294125 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:55.294308 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:55.295654 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:55.295730 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:55.297033 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:55.297072 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:55.298404 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:55.298467 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:55.298875 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:55.299802 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:06:55.299832 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:55.299921 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:06:55.300072 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:55.300748 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:55.300790 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:55.300828 (+ 38us) client_negotiation.cc:769] Sending connection context
0502 14:06:55.301030 (+ 202us) client_negotiation.cc:240] Negotiation successful
0502 14:06:55.301372 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"thread_start_us":1423,"threads_started":1}
I20240502 14:06:55.304991 4015 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44691
I20240502 14:06:55.305459 4015 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:55.306823 4015 heartbeater.cc:511] Master 127.27.174.126:44691 requested a full tablet report, sending...
I20240502 14:06:55.309541 3827 ts_manager.cc:194] Registered new tserver with Master: 89f033f1cb34432f8a1917cc651e4cb0 (127.27.174.65:43413)
I20240502 14:06:55.311548 3827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:39527
W20240502 14:06:55.584132 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:55.584601 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:55.585086 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:55.614342 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:55.614602 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:55.615396 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:06:55.648746 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:55.649876 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:55.651417 4019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:55.668195 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:56.315331 4015 heartbeater.cc:503] Master 127.27.174.126:44691 was elected leader, sending a full tablet report...
W20240502 14:06:55.668745 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:55.669202 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:55.669896 4019 server_base.cc:1034] running on GCE node
I20240502 14:06:56.713390 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:56.715665 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:56.716992 4019 hybrid_clock.cc:648] HybridClock initialized: now 1714658816716957 us; error 55 us; skew 500 ppm
I20240502 14:06:56.717764 4019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:56.729598 4019 webserver.cc:460] Webserver started at http://127.27.174.66:34863/ using document root <none> and password file <none>
I20240502 14:06:56.730543 4019 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:56.730753 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:56.731209 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:56.736245 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "b2df9803556943c2a0f8f3a55af3bbb4"
format_stamp: "Formatted at 2024-05-02 14:06:56 on dist-test-slave-dlb8"
server_key: "703c02553f9b1e3c5ddf22a81e3affc0"
server_key_iv: "ca3b5f4c6651b0ff3fd1bbf72636fadb"
server_key_version: "encryptionkey@0"
I20240502 14:06:56.737355 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "b2df9803556943c2a0f8f3a55af3bbb4"
format_stamp: "Formatted at 2024-05-02 14:06:56 on dist-test-slave-dlb8"
server_key: "703c02553f9b1e3c5ddf22a81e3affc0"
server_key_iv: "ca3b5f4c6651b0ff3fd1bbf72636fadb"
server_key_version: "encryptionkey@0"
I20240502 14:06:56.744112 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20240502 14:06:56.749593 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:56.750622 4019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:06:56.750916 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "b2df9803556943c2a0f8f3a55af3bbb4"
format_stamp: "Formatted at 2024-05-02 14:06:56 on dist-test-slave-dlb8"
server_key: "703c02553f9b1e3c5ddf22a81e3affc0"
server_key_iv: "ca3b5f4c6651b0ff3fd1bbf72636fadb"
server_key_version: "encryptionkey@0"
I20240502 14:06:56.751260 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:56.802994 4019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:56.804355 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:56.804775 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:56.807175 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:56.811086 4019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:56.811276 4019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:56.811537 4019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:56.811687 4019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:56.945611 4019 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41919
I20240502 14:06:56.945688 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41919 every 8 connection(s)
I20240502 14:06:56.948035 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:06:56.952297 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4019
I20240502 14:06:56.952800 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:06:56.954023 28345 external_mini_cluster.cc:1428] Setting key 5a16287f15b1341677f508823410d5ea
I20240502 14:06:56.958858 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--builtin_ntp_servers=127.27.174.84:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240502 14:06:56.968248 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:56.952042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45465 (local address 127.27.174.126:44691)
0502 14:06:56.953538 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:56.953569 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:56.955264 (+ 1695us) server_negotiation.cc:407] Connection header received
0502 14:06:56.959938 (+ 4674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:56.959963 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:56.960126 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:56.960269 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:56.961933 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:56.962840 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:56.964109 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:56.964644 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:56.965624 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:56.965691 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:56.965707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:06:56.965747 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:56.965827 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:06:56.965860 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:56.965974 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:56.966309 (+ 335us) server_negotiation.cc:299] Negotiation successful
0502 14:06:56.966494 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":501,"threads_started":1}
I20240502 14:06:56.968886 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:56.952190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44691 (local address 127.27.174.66:45465)
0502 14:06:56.955048 (+ 2858us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:56.955106 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:56.959630 (+ 4524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:56.960576 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:56.960605 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:56.960730 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:56.961665 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:56.961709 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:56.963087 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:56.963107 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:56.963902 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:56.963934 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:56.964256 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:56.965233 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:06:56.965248 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:56.965302 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:06:56.965398 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:56.966032 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:56.966073 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:56.966091 (+ 18us) client_negotiation.cc:769] Sending connection context
0502 14:06:56.966247 (+ 156us) client_negotiation.cc:240] Negotiation successful
0502 14:06:56.966547 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2062,"thread_start_us":1631,"threads_started":1}
I20240502 14:06:56.970790 4148 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44691
I20240502 14:06:56.971148 4148 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:56.972097 4148 heartbeater.cc:511] Master 127.27.174.126:44691 requested a full tablet report, sending...
I20240502 14:06:56.974386 3827 ts_manager.cc:194] Registered new tserver with Master: b2df9803556943c2a0f8f3a55af3bbb4 (127.27.174.66:41919)
I20240502 14:06:56.976270 3827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:45465
W20240502 14:06:57.242026 4152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:57.242506 4152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:57.242935 4152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:57.272223 4152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:57.272437 4152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:57.273097 4152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:06:57.311085 4152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:57.312201 4152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:57.313666 4152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:57.329978 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:57.979841 4148 heartbeater.cc:503] Master 127.27.174.126:44691 was elected leader, sending a full tablet report...
W20240502 14:06:57.330138 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:57.330183 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:57.331224 4152 server_base.cc:1034] running on GCE node
I20240502 14:06:58.401585 4152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:06:58.404309 4152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:06:58.405680 4152 hybrid_clock.cc:648] HybridClock initialized: now 1714658818405652 us; error 35 us; skew 500 ppm
I20240502 14:06:58.406555 4152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:58.413017 4152 webserver.cc:460] Webserver started at http://127.27.174.67:41401/ using document root <none> and password file <none>
I20240502 14:06:58.413925 4152 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:58.414168 4152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:58.414655 4152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:58.419649 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "1e5883a495fb4b4fae96a15ca3eda054"
format_stamp: "Formatted at 2024-05-02 14:06:58 on dist-test-slave-dlb8"
server_key: "2e4ed3c50aa8cec534a6dd6cb6e1ce0d"
server_key_iv: "2fc21e18d88d43ad0aed89a9d2a9d47d"
server_key_version: "encryptionkey@0"
I20240502 14:06:58.420782 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "1e5883a495fb4b4fae96a15ca3eda054"
format_stamp: "Formatted at 2024-05-02 14:06:58 on dist-test-slave-dlb8"
server_key: "2e4ed3c50aa8cec534a6dd6cb6e1ce0d"
server_key_iv: "2fc21e18d88d43ad0aed89a9d2a9d47d"
server_key_version: "encryptionkey@0"
I20240502 14:06:58.427641 4152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20240502 14:06:58.432968 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.433893 4152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:06:58.434226 4152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "1e5883a495fb4b4fae96a15ca3eda054"
format_stamp: "Formatted at 2024-05-02 14:06:58 on dist-test-slave-dlb8"
server_key: "2e4ed3c50aa8cec534a6dd6cb6e1ce0d"
server_key_iv: "2fc21e18d88d43ad0aed89a9d2a9d47d"
server_key_version: "encryptionkey@0"
I20240502 14:06:58.434588 4152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:58.482748 4152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:58.484144 4152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:06:58.484555 4152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:58.486944 4152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:58.490875 4152 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:58.491080 4152 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.491333 4152 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:58.491498 4152 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.619861 4152 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:43341
I20240502 14:06:58.619961 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:43341 every 8 connection(s)
I20240502 14:06:58.622661 4152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:06:58.632573 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4152
I20240502 14:06:58.633200 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:06:58.634392 28345 external_mini_cluster.cc:1428] Setting key 0464f9ef2082e4ef1e8cf7469ccbe427
I20240502 14:06:58.643105 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:58.627004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:35443 (local address 127.27.174.126:44691)
0502 14:06:58.628072 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:58.628110 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:58.629740 (+ 1630us) server_negotiation.cc:407] Connection header received
0502 14:06:58.634539 (+ 4799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:58.634563 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:58.634766 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:58.634940 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:58.636877 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:58.638195 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:06:58.639670 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:06:58.640083 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:58.641096 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:06:58.641168 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:06:58.641185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:06:58.641236 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:06:58.641305 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:06:58.641341 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:06:58.641452 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0502 14:06:58.641767 (+ 315us) server_negotiation.cc:299] Negotiation successful
0502 14:06:58.641908 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":399,"threads_started":1}
I20240502 14:06:58.643690 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:06:58.627350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:44691 (local address 127.27.174.67:35443)
0502 14:06:58.629516 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:58.629580 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:58.634187 (+ 4607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:58.635238 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:06:58.635277 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:06:58.635428 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:06:58.636598 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:58.636671 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:58.638488 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:06:58.638510 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:06:58.639425 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:06:58.639479 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:06:58.639886 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:06:58.640721 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:06:58.640736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:06:58.640790 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:06:58.640887 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:06:58.641526 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:06:58.641550 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:06:58.641569 (+ 19us) client_negotiation.cc:769] Sending connection context
0502 14:06:58.641725 (+ 156us) client_negotiation.cc:240] Negotiation successful
0502 14:06:58.641974 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":1099,"threads_started":1}
I20240502 14:06:58.645210 4282 heartbeater.cc:347] Connected to a master server at 127.27.174.126:44691
I20240502 14:06:58.645596 4282 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:58.646587 4282 heartbeater.cc:511] Master 127.27.174.126:44691 requested a full tablet report, sending...
I20240502 14:06:58.648489 3827 ts_manager.cc:194] Registered new tserver with Master: 1e5883a495fb4b4fae96a15ca3eda054 (127.27.174.67:43341)
I20240502 14:06:58.649811 3827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:35443
I20240502 14:06:58.653924 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240502 14:06:58.673571 4284 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:58.669464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.27.174.126:44691)
0502 14:06:58.669975 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0502 14:06:58.670002 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:06:58.671055 (+ 1053us) server_negotiation.cc:407] Connection header received
0502 14:06:58.671450 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:06:58.671468 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:06:58.671635 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:06:58.671787 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:06:58.672545 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56850: BlockingRecv error: recv got EOF from 127.0.0.1:56850 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20240502 14:06:58.673671 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3886
I20240502 14:06:58.697948 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4019
I20240502 14:06:58.720896 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4152
I20240502 14:06:58.741672 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 3794
2024-05-02T14:06:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (6985 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-02T14:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:06:58Z Disabled control of system clock
I20240502 14:06:58.823637 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34487
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240502 14:06:59.107473 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:06:59.108069 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:06:59.108613 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:06:59.136616 4294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:06:59.136891 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:06:59.137109 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:06:59.137331 4294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:06:59.137552 4294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:06:59.170161 4294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:06:59.171485 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:06:59.173132 4294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:59.188504 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:59.189976 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:59.188575 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:00.242412 4302 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:07:00.242460 4294 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:07:00.245978 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:00.248934 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:00.250341 4294 hybrid_clock.cc:648] HybridClock initialized: now 1714658820250320 us; error 46 us; skew 500 ppm
I20240502 14:07:00.251111 4294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:07:00.257967 4294 webserver.cc:460] Webserver started at http://127.27.174.126:42429/ using document root <none> and password file <none>
I20240502 14:07:00.258888 4294 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:00.259056 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:00.259476 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:00.264628 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "6a90e765a9b643ad9c78ad46b87d7f07"
format_stamp: "Formatted at 2024-05-02 14:07:00 on dist-test-slave-dlb8"
server_key: "5e0bacc1a763161672e6bf1b4f94af96"
server_key_iv: "584a1b6f934077c97c65771ee31731d6"
server_key_version: "encryptionkey@0"
I20240502 14:07:00.265673 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "6a90e765a9b643ad9c78ad46b87d7f07"
format_stamp: "Formatted at 2024-05-02 14:07:00 on dist-test-slave-dlb8"
server_key: "5e0bacc1a763161672e6bf1b4f94af96"
server_key_iv: "584a1b6f934077c97c65771ee31731d6"
server_key_version: "encryptionkey@0"
I20240502 14:07:00.272351 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:07:00.277349 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:00.278239 4294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:07:00.278513 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "6a90e765a9b643ad9c78ad46b87d7f07"
format_stamp: "Formatted at 2024-05-02 14:07:00 on dist-test-slave-dlb8"
server_key: "5e0bacc1a763161672e6bf1b4f94af96"
server_key_iv: "584a1b6f934077c97c65771ee31731d6"
server_key_version: "encryptionkey@0"
I20240502 14:07:00.278805 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:00.315836 4294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:00.317286 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:00.317713 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:00.384630 4294 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34487
I20240502 14:07:00.384711 4361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34487 every 8 connection(s)
I20240502 14:07:00.387290 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:07:00.392616 4362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:07:00.395846 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4294
I20240502 14:07:00.396279 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:07:00.397337 28345 external_mini_cluster.cc:1428] Setting key 742186eb8d493c3c58cc953165be85bc
I20240502 14:07:00.413724 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:00.399500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.27.174.126:34487)
0502 14:07:00.402387 (+ 2887us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:00.402423 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:00.402498 (+ 75us) server_negotiation.cc:407] Connection header received
0502 14:07:00.408053 (+ 5555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:00.408106 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:00.408575 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:00.408997 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:00.410605 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:00.410638 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:00.410681 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:07:00.410757 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:00.410871 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:07:00.410934 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:00.411201 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:00.411353 (+ 152us) server_negotiation.cc:299] Negotiation successful
0502 14:07:00.411599 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":1594,"threads_started":1}
I20240502 14:07:00.416199 4362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07: Bootstrap starting.
I20240502 14:07:00.421103 4362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:00.423084 4362 log.cc:824] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:00.427461 4362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07: No bootstrap required, opened a new log
I20240502 14:07:00.446494 4362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } }
I20240502 14:07:00.447120 4362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:00.447309 4362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90e765a9b643ad9c78ad46b87d7f07, State: Initialized, Role: FOLLOWER
I20240502 14:07:00.447886 4362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } }
I20240502 14:07:00.448318 4362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:07:00.448524 4362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:07:00.448747 4362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:00.453377 4362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } }
I20240502 14:07:00.453961 4362 leader_election.cc:302] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a90e765a9b643ad9c78ad46b87d7f07; no voters:
I20240502 14:07:00.455453 4362 leader_election.cc:288] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:07:00.456158 4367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:07:00.458141 4367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [term 1 LEADER]: Becoming Leader. State: Replica: 6a90e765a9b643ad9c78ad46b87d7f07, State: Running, Role: LEADER
I20240502 14:07:00.458735 4367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } }
I20240502 14:07:00.459108 4362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:07:00.468137 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } } }
I20240502 14:07:00.468008 4369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a90e765a9b643ad9c78ad46b87d7f07. Latest consensus state: current_term: 1 leader_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a90e765a9b643ad9c78ad46b87d7f07" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34487 } } }
I20240502 14:07:00.469067 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:00.469226 4369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:00.476401 4375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:07:00.485874 4375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:07:00.500578 4375 catalog_manager.cc:1347] Generated new cluster ID: 8b1ae3eeaa444edea674b267fda41afa
I20240502 14:07:00.500876 4375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:07:00.525894 4375 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:07:00.526427 4375 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:07:00.543202 4375 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6a90e765a9b643ad9c78ad46b87d7f07: Generated new TSK 0
I20240502 14:07:00.543943 4375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:07:00.556761 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--builtin_ntp_servers=127.27.174.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240502 14:07:00.878129 4386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:00.878603 4386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:00.879127 4386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:00.907193 4386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:00.907433 4386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:00.908257 4386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:07:00.942531 4386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:00.944031 4386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:00.945605 4386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:00.962819 4393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:00.964525 4392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:00.963887 4395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:02.039232 4394 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:07:02.039541 4386 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:07:02.043804 4386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:02.046514 4386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:02.047925 4386 hybrid_clock.cc:648] HybridClock initialized: now 1714658822047857 us; error 80 us; skew 500 ppm
I20240502 14:07:02.048729 4386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:07:02.059652 4386 webserver.cc:460] Webserver started at http://127.27.174.65:45173/ using document root <none> and password file <none>
I20240502 14:07:02.060703 4386 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:02.060940 4386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:02.061380 4386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:02.066468 4386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "b51aa9fd43ac4539b1c4e2394bffeb04"
format_stamp: "Formatted at 2024-05-02 14:07:02 on dist-test-slave-dlb8"
server_key: "68c2c794269fdf68286c61ed3a3334e9"
server_key_iv: "7649e6ff123c94deef9ca3e55eb39fe2"
server_key_version: "encryptionkey@0"
I20240502 14:07:02.067579 4386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "b51aa9fd43ac4539b1c4e2394bffeb04"
format_stamp: "Formatted at 2024-05-02 14:07:02 on dist-test-slave-dlb8"
server_key: "68c2c794269fdf68286c61ed3a3334e9"
server_key_iv: "7649e6ff123c94deef9ca3e55eb39fe2"
server_key_version: "encryptionkey@0"
I20240502 14:07:02.074764 4386 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20240502 14:07:02.080562 4402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:02.081622 4386 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20240502 14:07:02.081930 4386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "b51aa9fd43ac4539b1c4e2394bffeb04"
format_stamp: "Formatted at 2024-05-02 14:07:02 on dist-test-slave-dlb8"
server_key: "68c2c794269fdf68286c61ed3a3334e9"
server_key_iv: "7649e6ff123c94deef9ca3e55eb39fe2"
server_key_version: "encryptionkey@0"
I20240502 14:07:02.082329 4386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:02.120328 4386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:02.121837 4386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:02.122291 4386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:02.125247 4386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:02.129103 4386 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:02.129277 4386 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:02.129528 4386 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:02.129668 4386 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:02.258258 4386 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:43769
I20240502 14:07:02.258356 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:43769 every 8 connection(s)
I20240502 14:07:02.260603 4386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:07:02.264794 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4386
I20240502 14:07:02.265288 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:07:02.266682 28345 external_mini_cluster.cc:1428] Setting key 42e8edbe0cb5f54202464bc710191ec3
I20240502 14:07:02.271884 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--builtin_ntp_servers=127.27.174.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20240502 14:07:02.277175 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:02.264771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34487 (local address 127.27.174.65:39115)
0502 14:07:02.267661 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:02.267750 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:02.272717 (+ 4967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:02.273640 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:02.273661 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:02.273766 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:02.274498 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:07:02.274514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:02.274581 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:07:02.274732 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:02.275426 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:02.275444 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:02.275460 (+ 16us) client_negotiation.cc:769] Sending connection context
0502 14:07:02.275671 (+ 211us) client_negotiation.cc:240] Negotiation successful
0502 14:07:02.275870 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"thread_start_us":1600,"threads_started":1}
I20240502 14:07:02.277443 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:02.264498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39115 (local address 127.27.174.126:34487)
0502 14:07:02.266275 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:02.266319 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:02.267909 (+ 1590us) server_negotiation.cc:407] Connection header received
0502 14:07:02.273040 (+ 5131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:02.273075 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:02.273270 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:02.273462 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:02.274965 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:02.275002 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:02.275043 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:07:02.275112 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:02.275218 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:07:02.275274 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:02.275482 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:02.275676 (+ 194us) server_negotiation.cc:299] Negotiation successful
0502 14:07:02.275805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":498,"threads_started":1}
I20240502 14:07:02.279258 4515 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34487
I20240502 14:07:02.279601 4515 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:02.280277 4515 heartbeater.cc:511] Master 127.27.174.126:34487 requested a full tablet report, sending...
I20240502 14:07:02.282658 4327 ts_manager.cc:194] Registered new tserver with Master: b51aa9fd43ac4539b1c4e2394bffeb04 (127.27.174.65:43769)
W20240502 14:07:02.559281 4519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:02.559770 4519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:02.560452 4519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:02.590004 4519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:02.590298 4519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:02.591252 4519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:07:02.624711 4519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:02.625929 4519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:02.627559 4519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:02.643577 4526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:03.285434 4515 heartbeater.cc:503] Master 127.27.174.126:34487 was elected leader, sending a full tablet report...
I20240502 14:07:02.645968 4519 server_base.cc:1034] running on GCE node
W20240502 14:07:02.644589 4525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:02.644140 4528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:03.668044 4519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:03.670285 4519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:03.671633 4519 hybrid_clock.cc:648] HybridClock initialized: now 1714658823671610 us; error 45 us; skew 500 ppm
I20240502 14:07:03.672418 4519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:07:03.678635 4519 webserver.cc:460] Webserver started at http://127.27.174.66:42731/ using document root <none> and password file <none>
I20240502 14:07:03.679518 4519 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:03.679724 4519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:03.680159 4519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:03.685194 4519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c18f9a38c3c74c9eac3e93059b9a1368"
format_stamp: "Formatted at 2024-05-02 14:07:03 on dist-test-slave-dlb8"
server_key: "5bdae3bfbd26837422649186d124cd68"
server_key_iv: "b769d0edcf719186d9a29bf814bd6059"
server_key_version: "encryptionkey@0"
I20240502 14:07:03.686323 4519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c18f9a38c3c74c9eac3e93059b9a1368"
format_stamp: "Formatted at 2024-05-02 14:07:03 on dist-test-slave-dlb8"
server_key: "5bdae3bfbd26837422649186d124cd68"
server_key_iv: "b769d0edcf719186d9a29bf814bd6059"
server_key_version: "encryptionkey@0"
I20240502 14:07:03.692909 4519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:07:03.698072 4535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:03.699002 4519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:07:03.699282 4519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "c18f9a38c3c74c9eac3e93059b9a1368"
format_stamp: "Formatted at 2024-05-02 14:07:03 on dist-test-slave-dlb8"
server_key: "5bdae3bfbd26837422649186d124cd68"
server_key_iv: "b769d0edcf719186d9a29bf814bd6059"
server_key_version: "encryptionkey@0"
I20240502 14:07:03.699613 4519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:03.736858 4519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:03.738334 4519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:03.738770 4519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:03.741082 4519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:03.745007 4519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:03.745204 4519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:03.745471 4519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:03.745635 4519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:03.873370 4519 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45801
I20240502 14:07:03.873469 4647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45801 every 8 connection(s)
I20240502 14:07:03.875802 4519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:07:03.883895 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4519
I20240502 14:07:03.884464 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:07:03.885854 28345 external_mini_cluster.cc:1428] Setting key 71f0c995970ca95e084ebbacfb0ee742
I20240502 14:07:03.891733 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--builtin_ntp_servers=127.27.174.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20240502 14:07:03.893426 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:03.879641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:49947 (local address 127.27.174.126:34487)
0502 14:07:03.880805 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:03.880823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:03.883274 (+ 2451us) server_negotiation.cc:407] Connection header received
0502 14:07:03.889471 (+ 6197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:03.889508 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:03.889648 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:03.889775 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:03.891679 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:03.891699 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:03.891716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:07:03.891759 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:03.891829 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:07:03.891864 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:03.892050 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:03.892365 (+ 315us) server_negotiation.cc:299] Negotiation successful
0502 14:07:03.892459 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":466,"threads_started":1}
I20240502 14:07:03.894860 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:03.880035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34487 (local address 127.27.174.66:49947)
0502 14:07:03.883016 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:03.883079 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:03.889115 (+ 6036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:03.890078 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:03.890113 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:03.890271 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:03.891138 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:07:03.891168 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:03.891262 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:07:03.891422 (+ 160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:03.892032 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:03.892066 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:03.892095 (+ 29us) client_negotiation.cc:769] Sending connection context
0502 14:07:03.892434 (+ 339us) client_negotiation.cc:240] Negotiation successful
0502 14:07:03.892642 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"thread_start_us":1739,"threads_started":1}
I20240502 14:07:03.896522 4648 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34487
I20240502 14:07:03.896968 4648 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:03.897889 4648 heartbeater.cc:511] Master 127.27.174.126:34487 requested a full tablet report, sending...
I20240502 14:07:03.900223 4327 ts_manager.cc:194] Registered new tserver with Master: c18f9a38c3c74c9eac3e93059b9a1368 (127.27.174.66:45801)
W20240502 14:07:04.171406 4652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:04.171916 4652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:04.172520 4652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:04.200868 4652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:04.201185 4652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:04.202097 4652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:07:04.233110 4652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:04.234232 4652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:04.235703 4652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:04.250278 4661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:04.903380 4648 heartbeater.cc:503] Master 127.27.174.126:34487 was elected leader, sending a full tablet report...
W20240502 14:07:04.250420 4658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:04.251482 4652 server_base.cc:1034] running on GCE node
W20240502 14:07:04.250578 4659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:05.284334 4652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:05.286551 4652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:05.287892 4652 hybrid_clock.cc:648] HybridClock initialized: now 1714658825287858 us; error 55 us; skew 500 ppm
I20240502 14:07:05.288645 4652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:07:05.295143 4652 webserver.cc:460] Webserver started at http://127.27.174.67:39707/ using document root <none> and password file <none>
I20240502 14:07:05.295984 4652 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:05.296166 4652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:05.296586 4652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:05.301326 4652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "06cb6bd82f5c474c8ef4b76e2d0bd737"
format_stamp: "Formatted at 2024-05-02 14:07:05 on dist-test-slave-dlb8"
server_key: "2091a8c64eb2029c5b2db05b2d818de6"
server_key_iv: "9f78ce5d8a83b59a399673880c5bd2fd"
server_key_version: "encryptionkey@0"
I20240502 14:07:05.302528 4652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "06cb6bd82f5c474c8ef4b76e2d0bd737"
format_stamp: "Formatted at 2024-05-02 14:07:05 on dist-test-slave-dlb8"
server_key: "2091a8c64eb2029c5b2db05b2d818de6"
server_key_iv: "9f78ce5d8a83b59a399673880c5bd2fd"
server_key_version: "encryptionkey@0"
I20240502 14:07:05.309087 4652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 14:07:05.314486 4668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:05.315438 4652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:07:05.315716 4652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "06cb6bd82f5c474c8ef4b76e2d0bd737"
format_stamp: "Formatted at 2024-05-02 14:07:05 on dist-test-slave-dlb8"
server_key: "2091a8c64eb2029c5b2db05b2d818de6"
server_key_iv: "9f78ce5d8a83b59a399673880c5bd2fd"
server_key_version: "encryptionkey@0"
I20240502 14:07:05.316025 4652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:05.353785 4652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:05.355242 4652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:05.355690 4652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:05.357995 4652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:05.362003 4652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:05.362234 4652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:07:05.362478 4652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:05.362623 4652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:05.491375 4652 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:32979
I20240502 14:07:05.491482 4780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:32979 every 8 connection(s)
I20240502 14:07:05.493724 4652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:07:05.494560 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4652
I20240502 14:07:05.494933 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:07:05.495985 28345 external_mini_cluster.cc:1428] Setting key 0abb82ec649828b671079a7107aba7cc
I20240502 14:07:05.509387 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:05.498451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:52355 (local address 127.27.174.126:34487)
0502 14:07:05.500059 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:05.500091 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:05.501595 (+ 1504us) server_negotiation.cc:407] Connection header received
0502 14:07:05.506027 (+ 4432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:05.506077 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:05.506216 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:05.506360 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:05.507763 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:05.507781 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:05.507795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:07:05.507830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:05.507899 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:07:05.507931 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:05.508092 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:05.508335 (+ 243us) server_negotiation.cc:299] Negotiation successful
0502 14:07:05.508438 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":553,"threads_started":1}
I20240502 14:07:05.510219 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:05.498913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:34487 (local address 127.27.174.67:52355)
0502 14:07:05.501364 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:05.501449 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:05.505658 (+ 4209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:05.506554 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:05.506577 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:05.506686 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:05.507402 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:07:05.507416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:05.507470 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:07:05.507568 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:05.508077 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:05.508109 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:05.508139 (+ 30us) client_negotiation.cc:769] Sending connection context
0502 14:07:05.508420 (+ 281us) client_negotiation.cc:240] Negotiation successful
0502 14:07:05.508628 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"thread_start_us":1422,"threads_started":1}
I20240502 14:07:05.511749 4781 heartbeater.cc:347] Connected to a master server at 127.27.174.126:34487
I20240502 14:07:05.512102 4781 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:05.512890 4781 heartbeater.cc:511] Master 127.27.174.126:34487 requested a full tablet report, sending...
I20240502 14:07:05.514954 4327 ts_manager.cc:194] Registered new tserver with Master: 06cb6bd82f5c474c8ef4b76e2d0bd737 (127.27.174.67:32979)
I20240502 14:07:05.529436 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240502 14:07:05.544867 4783 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:07:05.540965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.27.174.126:34487)
0502 14:07:05.541518 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:05.541543 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:05.542579 (+ 1036us) server_negotiation.cc:407] Connection header received
0502 14:07:05.542915 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:05.542949 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:05.543073 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:05.543222 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:05.543829 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45370: BlockingRecv error: recv got EOF from 127.0.0.1:45370 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20240502 14:07:05.544924 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4386
I20240502 14:07:05.566941 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4519
I20240502 14:07:05.587807 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4652
I20240502 14:07:05.607653 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4294
2024-05-02T14:07:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (6867 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:07:05 dist-test-slave-dlb8 krb5kdc[4792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:07:05 dist-test-slave-dlb8 krb5kdc[4792](info): set up 2 sockets
May 02 14:07:05 dist-test-slave-dlb8 krb5kdc[4792](info): commencing operation
krb5kdc: starting...
W20240502 14:07:06.588339 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:07:06 dist-test-slave-dlb8 krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:07:06.798915 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41071
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:07:07.079470 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:07.080011 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:07.080516 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:07.108434 4805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:07:07.108788 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:07.109081 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:07.109328 4805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:07:07.109558 4805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:07:07.141701 4805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:07.142886 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:07.143865 4805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:07.157313 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:07.157824 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:07.159890 4805 server_base.cc:1034] running on GCE node
W20240502 14:07:07.158468 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:08.206459 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:08.209918 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:08.211355 4805 hybrid_clock.cc:648] HybridClock initialized: now 1714658828211314 us; error 61 us; skew 500 ppm
I20240502 14:07:08.212132 4805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:07:08 dist-test-slave-dlb8 krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658828, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:07:08.222541 4805 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:07:08.228150 4805 webserver.cc:460] Webserver started at http://127.27.174.126:43337/ using document root <none> and password file <none>
I20240502 14:07:08.229022 4805 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:08.229221 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:08.229622 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:08.234517 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "0ece0ab5568f4548b2fbc44fecc52cc6"
format_stamp: "Formatted at 2024-05-02 14:07:08 on dist-test-slave-dlb8"
server_key: "90afd305a33dfeb8e2b527ba9d874682"
server_key_iv: "6a77362f2e5be8730c84cf1988325406"
server_key_version: "encryptionkey@0"
I20240502 14:07:08.235608 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "0ece0ab5568f4548b2fbc44fecc52cc6"
format_stamp: "Formatted at 2024-05-02 14:07:08 on dist-test-slave-dlb8"
server_key: "90afd305a33dfeb8e2b527ba9d874682"
server_key_iv: "6a77362f2e5be8730c84cf1988325406"
server_key_version: "encryptionkey@0"
I20240502 14:07:08.242594 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:07:08.248019 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:08.249015 4805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:07:08.249346 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "0ece0ab5568f4548b2fbc44fecc52cc6"
format_stamp: "Formatted at 2024-05-02 14:07:08 on dist-test-slave-dlb8"
server_key: "90afd305a33dfeb8e2b527ba9d874682"
server_key_iv: "6a77362f2e5be8730c84cf1988325406"
server_key_version: "encryptionkey@0"
I20240502 14:07:08.249715 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:08.294893 4805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:08.299849 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:08.300297 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:08.366360 4805 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41071
I20240502 14:07:08.366431 4874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41071 every 8 connection(s)
I20240502 14:07:08.368744 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:07:08.370925 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4805
I20240502 14:07:08.371726 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:07:08.373461 28345 external_mini_cluster.cc:1428] Setting key ba85f92f8917d492c89f0d90b7ad6ca8
I20240502 14:07:08.375221 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:07:08 dist-test-slave-dlb8 krb5kdc[4792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:07:08.401479 4875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: Bootstrap starting.
I20240502 14:07:08.407886 4875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:08.410135 4875 log.cc:824] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:08.411733 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:08.376830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.27.174.126:41071)
0502 14:07:08.379460 (+ 2630us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:08.379484 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:08.379531 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:07:08.384541 (+ 5010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:08.384576 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:08.384881 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:08.385162 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:08.389033 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:08.390538 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:08.392343 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:08.392762 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:08.398688 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:08.398826 (+ 138us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:08.398868 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:08.399034 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:08.404697 (+ 5663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:08.405765 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:08.405796 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:08.405839 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:08.406093 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:08.406592 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:08.406616 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:08.406635 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:08.407359 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:08.407813 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:08.408311 (+ 498us) server_negotiation.cc:299] Negotiation successful
0502 14:07:08.408913 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":1735,"threads_started":1}
I20240502 14:07:08.415743 4875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: No bootstrap required, opened a new log
I20240502 14:07:08.435909 4875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } }
I20240502 14:07:08.436604 4875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:08.436853 4875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0ab5568f4548b2fbc44fecc52cc6, State: Initialized, Role: FOLLOWER
I20240502 14:07:08.437582 4875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } }
I20240502 14:07:08.438097 4875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:07:08.438354 4875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:07:08.438655 4875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:08.442533 4875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } }
I20240502 14:07:08.443226 4875 leader_election.cc:302] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ece0ab5568f4548b2fbc44fecc52cc6; no voters:
I20240502 14:07:08.444737 4875 leader_election.cc:288] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:07:08.445075 4880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:07:08.447032 4880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0ab5568f4548b2fbc44fecc52cc6, State: Running, Role: LEADER
I20240502 14:07:08.447741 4880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } }
I20240502 14:07:08.448828 4875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:07:08.457549 4882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ece0ab5568f4548b2fbc44fecc52cc6. Latest consensus state: current_term: 1 leader_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } } }
I20240502 14:07:08.458355 4882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:08.458168 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ece0ab5568f4548b2fbc44fecc52cc6" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41071 } } }
I20240502 14:07:08.458735 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:08.462610 4888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:07:08.475858 4888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:07:08.490257 4888 catalog_manager.cc:1347] Generated new cluster ID: 99da779ea6b048bc9247d930ed9206cd
I20240502 14:07:08.490578 4888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:07:08.519369 4888 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:07:08.520787 4888 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:07:08.532938 4888 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: Generated new TSK 0
I20240502 14:07:08.533818 4888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:07:08.590806 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:07:08.883682 4901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:08.884156 4901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:08.884724 4901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:08.911986 4901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:08.912338 4901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:08.913533 4901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:07:08.947369 4901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:08.948421 4901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:08.949374 4901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:08.966524 4908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:08.969147 4907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:08.969280 4901 server_base.cc:1034] running on GCE node
W20240502 14:07:08.968329 4910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:10.030539 4901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:10.033391 4901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:10.034857 4901 hybrid_clock.cc:648] HybridClock initialized: now 1714658830034824 us; error 36 us; skew 500 ppm
I20240502 14:07:10.035823 4901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:07:10 dist-test-slave-dlb8 krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658830, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:07:10.042750 4901 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:07:10.048139 4901 webserver.cc:460] Webserver started at http://127.27.174.65:43803/ using document root <none> and password file <none>
I20240502 14:07:10.049300 4901 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:10.049543 4901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:10.050067 4901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:10.056977 4901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "4570b22ebfcd4029b415db969edf70e6"
format_stamp: "Formatted at 2024-05-02 14:07:10 on dist-test-slave-dlb8"
server_key: "b73468ce0dee57ba4dcb28ec3a3c26e2"
server_key_iv: "c96573d6fad95d3d30ee3aa2dfcb1e74"
server_key_version: "encryptionkey@0"
I20240502 14:07:10.058457 4901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "4570b22ebfcd4029b415db969edf70e6"
format_stamp: "Formatted at 2024-05-02 14:07:10 on dist-test-slave-dlb8"
server_key: "b73468ce0dee57ba4dcb28ec3a3c26e2"
server_key_iv: "c96573d6fad95d3d30ee3aa2dfcb1e74"
server_key_version: "encryptionkey@0"
I20240502 14:07:10.067238 4901 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20240502 14:07:10.074116 4918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:10.075206 4901 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:07:10.075582 4901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "4570b22ebfcd4029b415db969edf70e6"
format_stamp: "Formatted at 2024-05-02 14:07:10 on dist-test-slave-dlb8"
server_key: "b73468ce0dee57ba4dcb28ec3a3c26e2"
server_key_iv: "c96573d6fad95d3d30ee3aa2dfcb1e74"
server_key_version: "encryptionkey@0"
I20240502 14:07:10.076035 4901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:10.140517 4901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:10.145560 4901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:10.145959 4901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:10.148264 4901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:10.152269 4901 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:10.152496 4901 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:10.152719 4901 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:10.152875 4901 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:10.281461 4901 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38683
I20240502 14:07:10.281565 5030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38683 every 8 connection(s)
I20240502 14:07:10.284010 4901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:07:10.286557 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 4901
I20240502 14:07:10.287218 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:07:10.288578 28345 external_mini_cluster.cc:1428] Setting key 9d1e42e427c47d9067e102c610160cc8
May 02 14:07:10 dist-test-slave-dlb8 krb5kdc[4792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658830, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:07:10.317322 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:10.288654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41071 (local address 127.27.174.65:48985)
0502 14:07:10.291184 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:10.291264 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:10.295839 (+ 4575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:10.296833 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:10.296877 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:10.297620 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:10.298647 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:10.298685 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:10.300224 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:10.300262 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:10.301556 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:10.301617 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:10.302110 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:10.303078 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:07:10.303191 (+ 113us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:10.307224 (+ 4033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:10.311935 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:10.311958 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:10.311980 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:10.312679 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:10.313204 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:10.313231 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:10.313245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:10.313426 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:10.314266 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:10.314297 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:10.314804 (+ 507us) client_negotiation.cc:769] Sending connection context
0502 14:07:10.315103 (+ 299us) client_negotiation.cc:240] Negotiation successful
0502 14:07:10.315487 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"thread_start_us":1478,"threads_started":1}
I20240502 14:07:10.318277 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:10.288386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:48985 (local address 127.27.174.126:41071)
0502 14:07:10.290183 (+ 1797us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:10.290211 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:10.291428 (+ 1217us) server_negotiation.cc:407] Connection header received
0502 14:07:10.296158 (+ 4730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:10.296190 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:10.296451 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:10.296642 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:10.298919 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:10.299985 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:10.301863 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:10.302275 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:10.307603 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:10.307675 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:10.307697 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:10.307791 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:10.311722 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:10.312871 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:10.312894 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:10.312919 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:10.313069 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:10.313588 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:10.313622 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:10.313650 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:10.314025 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:10.314276 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:10.315380 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0502 14:07:10.315829 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":701,"threads_started":1}
I20240502 14:07:10.320052 5031 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41071
I20240502 14:07:10.320420 5031 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:10.321516 5031 heartbeater.cc:511] Master 127.27.174.126:41071 requested a full tablet report, sending...
I20240502 14:07:10.324280 4840 ts_manager.cc:194] Registered new tserver with Master: 4570b22ebfcd4029b415db969edf70e6 (127.27.174.65:38683)
I20240502 14:07:10.326031 4840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:48985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:07:10.342016 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:07:10.639101 5037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:10.639616 5037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:10.640230 5037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:10.669467 5037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:10.669817 5037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:10.670665 5037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:07:10.706359 5037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:10.707505 5037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:10.708492 5037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:10.724469 5043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:11.329169 5031 heartbeater.cc:503] Master 127.27.174.126:41071 was elected leader, sending a full tablet report...
W20240502 14:07:10.724603 5046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:10.724887 5044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:10.726213 5037 server_base.cc:1034] running on GCE node
I20240502 14:07:11.761106 5037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:11.763450 5037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:11.764782 5037 hybrid_clock.cc:648] HybridClock initialized: now 1714658831764766 us; error 44 us; skew 500 ppm
I20240502 14:07:11.765520 5037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:07:11 dist-test-slave-dlb8 krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:07:11.771155 5037 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:07:11.776885 5037 webserver.cc:460] Webserver started at http://127.27.174.66:38699/ using document root <none> and password file <none>
I20240502 14:07:11.777738 5037 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:11.777930 5037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:11.778416 5037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:11.783576 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "c33f09fffd4a4b3a8e4b45725da555eb"
format_stamp: "Formatted at 2024-05-02 14:07:11 on dist-test-slave-dlb8"
server_key: "be4b2d907eb29a1886ea7990c1e6b022"
server_key_iv: "5710b5b11415784d8122da62e6ad65d4"
server_key_version: "encryptionkey@0"
I20240502 14:07:11.784574 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "c33f09fffd4a4b3a8e4b45725da555eb"
format_stamp: "Formatted at 2024-05-02 14:07:11 on dist-test-slave-dlb8"
server_key: "be4b2d907eb29a1886ea7990c1e6b022"
server_key_iv: "5710b5b11415784d8122da62e6ad65d4"
server_key_version: "encryptionkey@0"
I20240502 14:07:11.790993 5037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:07:11.796144 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:11.797107 5037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20240502 14:07:11.797396 5037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "c33f09fffd4a4b3a8e4b45725da555eb"
format_stamp: "Formatted at 2024-05-02 14:07:11 on dist-test-slave-dlb8"
server_key: "be4b2d907eb29a1886ea7990c1e6b022"
server_key_iv: "5710b5b11415784d8122da62e6ad65d4"
server_key_version: "encryptionkey@0"
I20240502 14:07:11.797732 5037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:11.869372 5037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:11.874536 5037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:11.874953 5037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:11.877323 5037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:11.880981 5037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:11.881176 5037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:11.881414 5037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:11.881562 5037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:12.006206 5037 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:46819
I20240502 14:07:12.006310 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:46819 every 8 connection(s)
I20240502 14:07:12.008541 5037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:07:12.017077 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 5037
I20240502 14:07:12.017517 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:07:12.018549 28345 external_mini_cluster.cc:1428] Setting key 946107ba5498b032acc053baebcc9a08
May 02 14:07:12 dist-test-slave-dlb8 krb5kdc[4792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:07:12.039891 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:12.014335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:53743 (local address 127.27.174.126:41071)
0502 14:07:12.015508 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:12.015522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:12.015560 (+ 38us) server_negotiation.cc:407] Connection header received
0502 14:07:12.020454 (+ 4894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:12.020483 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:12.020665 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:12.020815 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:12.023542 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:12.024928 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:12.026229 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:12.026822 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:12.030432 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:12.030497 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:12.030513 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:12.030600 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:12.034088 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:12.035454 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:12.035487 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:12.035500 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:12.035648 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:12.036432 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:12.036453 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:12.036467 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:12.036846 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:12.037018 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:12.038259 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0502 14:07:12.038594 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":501,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:07:12.041460 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:12.012682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41071 (local address 127.27.174.66:53743)
0502 14:07:12.015388 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:12.015452 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:12.020039 (+ 4587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:12.021105 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:12.021139 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:12.022280 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:12.023256 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:12.023296 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:12.025149 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:12.025170 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:12.025964 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:12.026006 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:12.026300 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:12.027084 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:07:12.027142 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:12.030149 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:12.034302 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:12.034339 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:12.034377 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:12.035271 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:12.035828 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:12.035869 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:12.035901 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:12.036204 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:12.037054 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:12.037097 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:12.037655 (+ 558us) client_negotiation.cc:769] Sending connection context
0502 14:07:12.038113 (+ 458us) client_negotiation.cc:240] Negotiation successful
0502 14:07:12.038719 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"thread_start_us":1476,"threads_started":1}
I20240502 14:07:12.043325 5167 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:07:12.043701 5167 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:12.044924 5167 heartbeater.cc:511] Master 127.27.174.126:41071 requested a full tablet report, sending...
I20240502 14:07:12.047363 4840 ts_manager.cc:194] Registered new tserver with Master: c33f09fffd4a4b3a8e4b45725da555eb (127.27.174.66:46819)
I20240502 14:07:12.048581 4840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:53743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:07:12.068567 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:07:12.362234 5173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:12.362748 5173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:12.363366 5173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:12.391893 5173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:12.392253 5173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:12.393149 5173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:07:12.426338 5173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:12.427501 5173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:12.428478 5173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:12.431411 4871 debug-util.cc:398] Leaking SignalData structure 0x7b08000af940 after lost signal to thread 4806
W20240502 14:07:12.446583 5180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:13.051903 5167 heartbeater.cc:503] Master 127.27.174.126:41071 was elected leader, sending a full tablet report...
W20240502 14:07:12.446679 5182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:12.448886 5173 server_base.cc:1034] running on GCE node
W20240502 14:07:12.447574 5179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:13.551052 5173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:13.556650 5173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:13.558081 5173 hybrid_clock.cc:648] HybridClock initialized: now 1714658833558006 us; error 73 us; skew 500 ppm
I20240502 14:07:13.558858 5173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:07:13 dist-test-slave-dlb8 krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658833, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:07:13.571182 5173 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:07:13.576723 5173 webserver.cc:460] Webserver started at http://127.27.174.67:39345/ using document root <none> and password file <none>
I20240502 14:07:13.577929 5173 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:13.578231 5173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:13.578832 5173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:07:13.586930 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "96b2c2d4dab941928f46fdfe08577f57"
format_stamp: "Formatted at 2024-05-02 14:07:13 on dist-test-slave-dlb8"
server_key: "9f4d195dad53d9bfa0d890d6834e38f7"
server_key_iv: "045f906417adcaab4657d49c6e2db105"
server_key_version: "encryptionkey@0"
I20240502 14:07:13.588471 5173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "96b2c2d4dab941928f46fdfe08577f57"
format_stamp: "Formatted at 2024-05-02 14:07:13 on dist-test-slave-dlb8"
server_key: "9f4d195dad53d9bfa0d890d6834e38f7"
server_key_iv: "045f906417adcaab4657d49c6e2db105"
server_key_version: "encryptionkey@0"
I20240502 14:07:13.598137 5173 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20240502 14:07:13.605763 5190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:13.607059 5173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:07:13.607506 5173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "96b2c2d4dab941928f46fdfe08577f57"
format_stamp: "Formatted at 2024-05-02 14:07:13 on dist-test-slave-dlb8"
server_key: "9f4d195dad53d9bfa0d890d6834e38f7"
server_key_iv: "045f906417adcaab4657d49c6e2db105"
server_key_version: "encryptionkey@0"
I20240502 14:07:13.608042 5173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:13.678329 5173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:13.686000 5173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:13.686645 5173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:13.689921 5173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:07:13.695647 5173 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:07:13.695957 5173 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:13.696293 5173 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:07:13.696497 5173 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:13.851975 5173 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:36421
I20240502 14:07:13.852090 5302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:36421 every 8 connection(s)
I20240502 14:07:13.855899 5173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:07:13.862151 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 5173
I20240502 14:07:13.862820 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:07:13.864626 28345 external_mini_cluster.cc:1428] Setting key b56733778779f3958af2bafca96412dd
May 02 14:07:13 dist-test-slave-dlb8 krb5kdc[4792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658833, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:07:13.896834 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:13.861012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56123 (local address 127.27.174.126:41071)
0502 14:07:13.862554 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:13.862585 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:13.871831 (+ 9246us) server_negotiation.cc:407] Connection header received
0502 14:07:13.876125 (+ 4294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:13.876160 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:13.876388 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:13.876567 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:13.878812 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.880286 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:13.881588 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.882667 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:13.887434 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:13.887498 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:13.887513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:13.887598 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:13.891071 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:13.892451 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:13.892474 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:13.892488 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:13.892615 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:13.893330 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:13.893346 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:13.893358 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:13.893689 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:13.893849 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:13.894951 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0502 14:07:13.895262 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":5485,"threads_started":1}
I20240502 14:07:13.898116 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:13.861380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41071 (local address 127.27.174.67:56123)
0502 14:07:13.871611 (+ 10231us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:13.871678 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:13.875785 (+ 4107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:13.876746 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:13.876768 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:13.877564 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:07:13.878569 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:13.878609 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.880503 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:13.880525 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:13.881337 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:13.881402 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.881732 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:13.883006 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:07:13.883110 (+ 104us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:07:13.887132 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:07:13.891270 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:13.891304 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:13.891339 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:13.892270 (+ 931us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:13.892772 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:07:13.892804 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:07:13.892832 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:07:13.893114 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:07:13.893852 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:07:13.893884 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:07:13.894425 (+ 541us) client_negotiation.cc:769] Sending connection context
0502 14:07:13.894821 (+ 396us) client_negotiation.cc:240] Negotiation successful
0502 14:07:13.895383 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9528,"thread_start_us":8977,"threads_started":1}
I20240502 14:07:13.900120 5303 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41071
I20240502 14:07:13.900460 5303 heartbeater.cc:464] Registering TS with master...
I20240502 14:07:13.901324 5303 heartbeater.cc:511] Master 127.27.174.126:41071 requested a full tablet report, sending...
I20240502 14:07:13.903678 4839 ts_manager.cc:194] Registered new tserver with Master: 96b2c2d4dab941928f46fdfe08577f57 (127.27.174.67:36421)
I20240502 14:07:13.905575 4839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:56123
I20240502 14:07:13.913847 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:07:13.951295 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:13.929768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.27.174.126:41071)
0502 14:07:13.930598 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:13.930623 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:13.931465 (+ 842us) server_negotiation.cc:407] Connection header received
0502 14:07:13.931772 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:13.931801 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:13.932061 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:13.932261 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:13.934579 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.935927 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:13.937256 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:13.937841 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:13.940007 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:13.940113 (+ 106us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:13.940163 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:13.940318 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:13.945405 (+ 5087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:13.946480 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:13.946515 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:13.946541 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:13.946735 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:13.947232 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:13.947261 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:13.947287 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:13.947824 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:13.948189 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:13.948652 (+ 463us) server_negotiation.cc:299] Negotiation successful
0502 14:07:13.949096 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20240502 14:07:13.957763 4839 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:07:13.978508 4839 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:07:14.054373 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.010664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.27.174.66:46819)
0502 14:07:14.015313 (+ 4649us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.015349 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.038971 (+ 23622us) server_negotiation.cc:407] Connection header received
0502 14:07:14.039338 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.039372 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.039732 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.039937 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:07:14.042617 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.044923 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.047502 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.048805 (+ 1303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.049052 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.052461 (+ 3409us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.052726 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3852,"thread_start_us":3868,"threads_started":1}
I20240502 14:07:14.061371 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.008960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:46819 (local address 127.0.0.1:50964)
0502 14:07:14.038603 (+ 29643us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.038832 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:14.039075 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:14.041134 (+ 2059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:14.041175 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:14.041281 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:07:14.042259 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.042319 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.045297 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.045335 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:14.047196 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.047247 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.050701 (+ 3454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.050833 (+ 132us) client_negotiation.cc:769] Sending connection context
0502 14:07:14.055235 (+ 4402us) client_negotiation.cc:240] Negotiation successful
0502 14:07:14.056097 (+ 862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28937,"thread_start_us":31652,"threads_started":1}
I20240502 14:07:14.063069 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.020566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38683 (local address 127.0.0.1:44782)
0502 14:07:14.040270 (+ 19704us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.040328 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:14.040544 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:14.049211 (+ 8667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:14.049246 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:14.049345 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:07:14.050291 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.050340 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.058592 (+ 8252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.058624 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:14.060513 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.060564 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.060866 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.060969 (+ 103us) client_negotiation.cc:769] Sending connection context
0502 14:07:14.061108 (+ 139us) client_negotiation.cc:240] Negotiation successful
0502 14:07:14.061336 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19122,"thread_start_us":16196,"threads_started":1}
I20240502 14:07:14.066888 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.020939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.27.174.65:38683)
0502 14:07:14.047061 (+ 26122us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.047107 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.047177 (+ 70us) server_negotiation.cc:407] Connection header received
0502 14:07:14.047436 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.047488 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.047921 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.048165 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:07:14.051119 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.058228 (+ 7109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.063566 (+ 5338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.064799 (+ 1233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.065030 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.065222 (+ 192us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.065448 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21892,"thread_start_us":21548,"threads_started":1}
I20240502 14:07:14.066223 5102 tablet_service.cc:1467] Processing CreateTablet for tablet a712b9bc94f3499d9e8785603064e9ff (DEFAULT_TABLE table=test-table [id=19c466a1be434b0283186118f956a48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:07:14.070307 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.007759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36421 (local address 127.0.0.1:40198)
0502 14:07:14.036125 (+ 28366us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.036224 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:14.036606 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:14.055510 (+ 18904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:14.055544 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:14.055652 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:07:14.060357 (+ 4705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.060417 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.063599 (+ 3182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.063632 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:14.067852 (+ 4220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.067901 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.068196 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.068282 (+ 86us) client_negotiation.cc:769] Sending connection context
0502 14:07:14.068414 (+ 132us) client_negotiation.cc:240] Negotiation successful
0502 14:07:14.068624 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27627,"thread_start_us":31651,"threads_started":1}
I20240502 14:07:14.071461 5102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a712b9bc94f3499d9e8785603064e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:07:14.072093 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.007992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.27.174.67:36421)
0502 14:07:14.054606 (+ 46614us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.054647 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.054692 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:07:14.054915 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.054948 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.055267 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.055466 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:07:14.060740 (+ 5274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.063322 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.068469 (+ 5147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.069781 (+ 1312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.069992 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.070223 (+ 231us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.070469 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45935,"thread_start_us":45545,"threads_started":1}
I20240502 14:07:14.071681 4966 tablet_service.cc:1467] Processing CreateTablet for tablet a712b9bc94f3499d9e8785603064e9ff (DEFAULT_TABLE table=test-table [id=19c466a1be434b0283186118f956a48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:07:14.075250 4966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a712b9bc94f3499d9e8785603064e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:07:14.076220 5238 tablet_service.cc:1467] Processing CreateTablet for tablet a712b9bc94f3499d9e8785603064e9ff (DEFAULT_TABLE table=test-table [id=19c466a1be434b0283186118f956a48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:07:14.077999 5238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a712b9bc94f3499d9e8785603064e9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:07:14.105137 5322 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Bootstrap starting.
I20240502 14:07:14.113987 5323 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Bootstrap starting.
I20240502 14:07:14.114125 5322 tablet_bootstrap.cc:654] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:14.116961 5322 log.cc:824] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:14.119014 5324 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Bootstrap starting.
I20240502 14:07:14.123106 5323 tablet_bootstrap.cc:654] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:14.125372 5323 log.cc:824] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:14.127198 5324 tablet_bootstrap.cc:654] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:14.130955 5324 log.cc:824] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:14.134896 5322 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: No bootstrap required, opened a new log
I20240502 14:07:14.135599 5322 ts_tablet_manager.cc:1367] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.014s
I20240502 14:07:14.139766 5323 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: No bootstrap required, opened a new log
I20240502 14:07:14.140322 5323 ts_tablet_manager.cc:1367] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.003s
I20240502 14:07:14.145861 5324 tablet_bootstrap.cc:492] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: No bootstrap required, opened a new log
I20240502 14:07:14.146432 5324 ts_tablet_manager.cc:1367] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.009s
I20240502 14:07:14.171118 5323 raft_consensus.cc:348] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.172219 5323 raft_consensus.cc:374] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:14.172562 5323 raft_consensus.cc:729] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b2c2d4dab941928f46fdfe08577f57, State: Initialized, Role: FOLLOWER
I20240502 14:07:14.173629 5323 consensus_queue.cc:260] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.175910 5324 raft_consensus.cc:348] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.176402 5322 raft_consensus.cc:348] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.177127 5324 raft_consensus.cc:374] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:14.177443 5322 raft_consensus.cc:374] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:14.177606 5324 raft_consensus.cc:729] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4570b22ebfcd4029b415db969edf70e6, State: Initialized, Role: FOLLOWER
I20240502 14:07:14.177753 5322 raft_consensus.cc:729] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33f09fffd4a4b3a8e4b45725da555eb, State: Initialized, Role: FOLLOWER
I20240502 14:07:14.178755 5322 consensus_queue.cc:260] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.178946 5324 consensus_queue.cc:260] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.183475 5303 heartbeater.cc:503] Master 127.27.174.126:41071 was elected leader, sending a full tablet report...
I20240502 14:07:14.184463 5323 ts_tablet_manager.cc:1398] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Time spent starting tablet: real 0.044s user 0.030s sys 0.011s
I20240502 14:07:14.187464 5324 ts_tablet_manager.cc:1398] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Time spent starting tablet: real 0.041s user 0.030s sys 0.007s
I20240502 14:07:14.190501 5322 ts_tablet_manager.cc:1398] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Time spent starting tablet: real 0.055s user 0.034s sys 0.007s
I20240502 14:07:14.202212 5330 raft_consensus.cc:482] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:14.202612 5330 raft_consensus.cc:504] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.204998 5330 leader_election.cc:288] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b2c2d4dab941928f46fdfe08577f57 (127.27.174.67:36421), c33f09fffd4a4b3a8e4b45725da555eb (127.27.174.66:46819)
I20240502 14:07:14.216411 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.205198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:36421 (local address 127.27.174.65:57487)
0502 14:07:14.207826 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.207879 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:14.208309 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:14.209449 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:14.209480 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:14.209607 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:07:14.210753 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.210808 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.212296 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.212324 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:14.213979 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.214029 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.214414 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.214488 (+ 74us) client_negotiation.cc:769] Sending connection context
0502 14:07:14.214610 (+ 122us) client_negotiation.cc:240] Negotiation successful
0502 14:07:14.214788 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"thread_start_us":2678,"threads_started":1}
I20240502 14:07:14.217062 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.207054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:57487 (local address 127.27.174.67:36421)
0502 14:07:14.207824 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.207853 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.207988 (+ 135us) server_negotiation.cc:407] Connection header received
0502 14:07:14.208630 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.208662 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.208912 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.209042 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:07:14.211031 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.211956 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.214327 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.215405 (+ 1078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.215546 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.215720 (+ 174us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.215904 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20240502 14:07:14.218693 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a712b9bc94f3499d9e8785603064e9ff" candidate_uuid: "4570b22ebfcd4029b415db969edf70e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b2c2d4dab941928f46fdfe08577f57" is_pre_election: true
I20240502 14:07:14.219570 5258 raft_consensus.cc:2454] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4570b22ebfcd4029b415db969edf70e6 in term 0.
I20240502 14:07:14.220041 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.208160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:53639 (local address 127.27.174.66:46819)
0502 14:07:14.209710 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.209738 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.210500 (+ 762us) server_negotiation.cc:407] Connection header received
0502 14:07:14.210804 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.210824 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.211020 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.211190 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:07:14.212758 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.213973 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.216430 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.217653 (+ 1223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.217793 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.218603 (+ 810us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.218820 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20240502 14:07:14.220412 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.205840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:46819 (local address 127.27.174.65:53639)
0502 14:07:14.210344 (+ 4504us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.210384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0502 14:07:14.210604 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:07:14.211251 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:07:14.211285 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:07:14.211388 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:07:14.212465 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.212520 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.214282 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.214316 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:07:14.216102 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:07:14.216180 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.218036 (+ 1856us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.218181 (+ 145us) client_negotiation.cc:769] Sending connection context
0502 14:07:14.218361 (+ 180us) client_negotiation.cc:240] Negotiation successful
0502 14:07:14.218589 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3899,"thread_start_us":3944,"threads_started":1}
I20240502 14:07:14.221207 4921 leader_election.cc:302] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4570b22ebfcd4029b415db969edf70e6, 96b2c2d4dab941928f46fdfe08577f57; no voters:
I20240502 14:07:14.221875 5330 raft_consensus.cc:2781] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:07:14.222275 5330 raft_consensus.cc:482] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:07:14.222612 5330 raft_consensus.cc:3037] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:14.223039 5122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a712b9bc94f3499d9e8785603064e9ff" candidate_uuid: "4570b22ebfcd4029b415db969edf70e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" is_pre_election: true
I20240502 14:07:14.223937 5122 raft_consensus.cc:2454] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4570b22ebfcd4029b415db969edf70e6 in term 0.
I20240502 14:07:14.227128 5330 raft_consensus.cc:504] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.228756 5330 leader_election.cc:288] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [CANDIDATE]: Term 1 election: Requested vote from peers 96b2c2d4dab941928f46fdfe08577f57 (127.27.174.67:36421), c33f09fffd4a4b3a8e4b45725da555eb (127.27.174.66:46819)
I20240502 14:07:14.229393 5122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a712b9bc94f3499d9e8785603064e9ff" candidate_uuid: "4570b22ebfcd4029b415db969edf70e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33f09fffd4a4b3a8e4b45725da555eb"
I20240502 14:07:14.229362 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a712b9bc94f3499d9e8785603064e9ff" candidate_uuid: "4570b22ebfcd4029b415db969edf70e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b2c2d4dab941928f46fdfe08577f57"
I20240502 14:07:14.229874 5122 raft_consensus.cc:3037] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:14.229969 5258 raft_consensus.cc:3037] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:14.234442 5122 raft_consensus.cc:2454] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4570b22ebfcd4029b415db969edf70e6 in term 1.
I20240502 14:07:14.235284 4920 leader_election.cc:302] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4570b22ebfcd4029b415db969edf70e6, c33f09fffd4a4b3a8e4b45725da555eb; no voters:
I20240502 14:07:14.235868 5330 raft_consensus.cc:2781] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:07:14.237339 5258 raft_consensus.cc:2454] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4570b22ebfcd4029b415db969edf70e6 in term 1.
I20240502 14:07:14.237875 5330 raft_consensus.cc:686] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4570b22ebfcd4029b415db969edf70e6, State: Running, Role: LEADER
I20240502 14:07:14.239171 5330 consensus_queue.cc:237] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } }
I20240502 14:07:14.251173 4839 catalog_manager.cc:5484] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4570b22ebfcd4029b415db969edf70e6 (127.27.174.65). New cstate: current_term: 1 leader_uuid: "4570b22ebfcd4029b415db969edf70e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4570b22ebfcd4029b415db969edf70e6" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38683 } health_report { overall_health: HEALTHY } } }
W20240502 14:07:14.264894 5168 tablet.cc:2369] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:07:14.292742 5032 tablet.cc:2369] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:07:14.309355 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.302260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.27.174.65:38683)
0502 14:07:14.302985 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:14.303014 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:14.303085 (+ 71us) server_negotiation.cc:407] Connection header received
0502 14:07:14.303523 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:14.303541 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:14.303774 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:14.303927 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:07:14.304712 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.305476 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:14.306709 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:14.307167 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:14.307332 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:07:14.307790 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:07:14.307946 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:14.308184 (+ 238us) server_negotiation.cc:299] Negotiation successful
0502 14:07:14.308335 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20240502 14:07:14.322222 5122 raft_consensus.cc:1261] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 1 FOLLOWER]: Refusing update from remote peer 4570b22ebfcd4029b415db969edf70e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:07:14.322196 5258 raft_consensus.cc:1261] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 1 FOLLOWER]: Refusing update from remote peer 4570b22ebfcd4029b415db969edf70e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:07:14.324247 5333 consensus_queue.cc:1035] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b2c2d4dab941928f46fdfe08577f57" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:07:14.325202 5330 consensus_queue.cc:1035] T a712b9bc94f3499d9e8785603064e9ff P 4570b22ebfcd4029b415db969edf70e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33f09fffd4a4b3a8e4b45725da555eb" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 46819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:07:14.356511 5336 mvcc.cc:205] Tried to move back new op lower bound from 7023242585367773184 to 7023242585050554368. Current Snapshot: MvccSnapshot[applied={T|T < 7023242585367773184}]
W20240502 14:07:14.367275 5304 tablet.cc:2369] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:07:14.401410 5337 mvcc.cc:205] Tried to move back new op lower bound from 7023242585367773184 to 7023242585050554368. Current Snapshot: MvccSnapshot[applied={T|T < 7023242585367773184}]
I20240502 14:07:14.416827 4839 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:07:14.417426 4839 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:07:14.425514 4839 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0ece0ab5568f4548b2fbc44fecc52cc6: Sending DeleteTablet for 3 replicas of tablet a712b9bc94f3499d9e8785603064e9ff
I20240502 14:07:14.427215 5238 tablet_service.cc:1514] Processing DeleteTablet for tablet a712b9bc94f3499d9e8785603064e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:07:14 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:40198
I20240502 14:07:14.429580 4966 tablet_service.cc:1514] Processing DeleteTablet for tablet a712b9bc94f3499d9e8785603064e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:07:14 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:44782
I20240502 14:07:14.430004 5346 tablet_replica.cc:330] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: stopping tablet replica
I20240502 14:07:14.431062 5346 raft_consensus.cc:2229] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:07:14.431854 5346 raft_consensus.cc:2258] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:07:14.432365 5102 tablet_service.cc:1514] Processing DeleteTablet for tablet a712b9bc94f3499d9e8785603064e9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:07:14 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:50964
I20240502 14:07:14.438445 5346 ts_tablet_manager.cc:1875] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:07:14.440308 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4901
I20240502 14:07:14.451581 5348 tablet_replica.cc:330] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: stopping tablet replica
I20240502 14:07:14.453186 5348 raft_consensus.cc:2229] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:07:14.453922 5348 raft_consensus.cc:2258] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:07:14.454326 5346 ts_tablet_manager.cc:1888] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:07:14.454735 5346 log.cc:1196] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/a712b9bc94f3499d9e8785603064e9ff
I20240502 14:07:14.455541 5346 ts_tablet_manager.cc:1909] T a712b9bc94f3499d9e8785603064e9ff P 96b2c2d4dab941928f46fdfe08577f57: Deleting consensus metadata
I20240502 14:07:14.457408 5348 ts_tablet_manager.cc:1875] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:07:14.458333 4826 catalog_manager.cc:4830] TS 96b2c2d4dab941928f46fdfe08577f57 (127.27.174.67:36421): tablet a712b9bc94f3499d9e8785603064e9ff (table test-table [id=19c466a1be434b0283186118f956a48c]) successfully deleted
I20240502 14:07:14.467561 5348 ts_tablet_manager.cc:1888] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:07:14.467859 5348 log.cc:1196] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/a712b9bc94f3499d9e8785603064e9ff
I20240502 14:07:14.468463 5348 ts_tablet_manager.cc:1909] T a712b9bc94f3499d9e8785603064e9ff P c33f09fffd4a4b3a8e4b45725da555eb: Deleting consensus metadata
I20240502 14:07:14.470796 4825 catalog_manager.cc:4830] TS c33f09fffd4a4b3a8e4b45725da555eb (127.27.174.66:46819): tablet a712b9bc94f3499d9e8785603064e9ff (table test-table [id=19c466a1be434b0283186118f956a48c]) successfully deleted
W20240502 14:07:14.479450 4824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:38683 (error 108)
I20240502 14:07:14.480372 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 5037
I20240502 14:07:14.491310 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:14.489808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38683 (local address 127.0.0.1:44798)
0502 14:07:14.490526 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0502 14:07:14.490779 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20240502 14:07:14.491858 4824 catalog_manager.cc:4573] TS 4570b22ebfcd4029b415db969edf70e6 (127.27.174.65:38683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a712b9bc94f3499d9e8785603064e9ff: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38683: connect: Connection refused (error 111)
I20240502 14:07:14.508419 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 5173
I20240502 14:07:14.537457 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 4805
2024-05-02T14:07:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (8933 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:07:14 dist-test-slave-dlb8 krb5kdc[5352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:07:14 dist-test-slave-dlb8 krb5kdc[5352](info): set up 2 sockets
May 02 14:07:14 dist-test-slave-dlb8 krb5kdc[5352](info): commencing operation
krb5kdc: starting...
W20240502 14:07:15.532449 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.009s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:07:15 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:07:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
I20240502 14:07:15.819607 28345 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/postgres -l logfile start
2024-05-02 14:07:18.224 UTC [5383] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-02 14:07:18.224 UTC [5383] LOG: listening on IPv4 address "127.27.174.84", port 52637
2024-05-02 14:07:18.227 UTC [5383] LOG: listening on Unix socket "/tmp/.s.PGSQL.52637"
2024-05-02 14:07:18.248 UTC [5386] LOG: database system was shut down at 2024-05-02 14:07:16 UTC
2024-05-02 14:07:18.252 UTC [5383] LOG: database system is ready to accept connections
I20240502 14:07:19.103897 28345 mini_postgres.cc:96] Postgres bound to 52637
2024-05-02 14:07:19.119 UTC [5394] FATAL: database "slave" does not exist
127.27.174.84:52637 - accepting connections
I20240502 14:07:19.119784 28345 mini_ranger.cc:161] Starting Ranger...
I20240502 14:07:19.142678 28345 mini_ranger.cc:84] Created miniranger Postgres user
I20240502 14:07:19.617687 28345 mini_ranger.cc:87] Created ranger Postgres database
I20240502 14:07:19.617955 28345 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin
2024-05-02 14:07:19,836 [I] DB FLAVOR :POSTGRES
2024-05-02 14:07:19,837 [I] --------- Verifying Ranger DB connection ---------
2024-05-02 14:07:19,837 [I] Checking connection..
2024-05-02 14:07:19,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-02 14:07:20,065 [I] Checking connection passed.
2024-05-02 14:07:20,065 [I] --------- Verifying version history table ---------
2024-05-02 14:07:20,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:07:20,306 [I] Table x_db_version_h does not exist in database ranger
2024-05-02 14:07:20,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:07:20,528 [I] Table x_db_version_h does not exist in database ranger
2024-05-02 14:07:20,529 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-02 14:07:20,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-02 14:07:20.757 UTC [5458] WARNING: there is no transaction in progress
2024-05-02 14:07:20,764 [I] create_dbversion_catalog.sql file imported successfully
2024-05-02 14:07:20,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:07:20,999 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-02 14:07:21,000 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-02 14:07:21,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-02 14:07:21,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-02 14:07:21,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dlb8.c.gcp-upstream.internal','N') ;"
2024-05-02 14:07:21,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-02 14:07:21,949 [I] Table x_portal_user does not exist in database ranger
2024-05-02 14:07:21,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-02 14:07:22,187 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-02 14:07:22,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-02 14:07:22,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-02 14:07:22,654 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-02 14:07:22,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-02 14:07:23.586 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.601 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.617 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.631 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.646 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.663 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.714 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.721 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.735 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.744 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.754 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.766 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.775 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.781 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.788 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:23.794 UTC [5566] WARNING: there is no transaction in progress
2024-05-02 14:07:24,404 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-02 14:07:24,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-02 14:07:24,639 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-02 14:07:24,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-02 14:07:24,864 [I] Table x_portal_user already exists in database 'ranger'
2024-05-02 14:07:24,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-02 14:07:25,096 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-02 14:07:25,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-02 14:07:25,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-02 14:07:25,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dlb8.c.gcp-upstream.internal';"
2024-05-02 14:07:25,801 [I] CORE_DB_SCHEMA import status has been updated
2024-05-02 14:07:25,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-02 14:07:26,038 [I] DB_PATCHES have already been applied
I20240502 14:07:26.044387 28345 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTLmxo8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240502 14:07:26.044610 28345 mini_ranger.cc:193] Using host: 127.27.174.84
I20240502 14:07:26.057380 28345 mini_ranger.cc:227] Ranger admin URL: http://127.27.174.84:57715
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp] .....
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp].
May 02, 2024 2:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240502 14:07:27.050871 28345 mini_ranger.cc:160] Time spent starting Ranger: real 7.931s user 0.001s sys 0.039s
May 02, 2024 2:07:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 02, 2024 2:07:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 02, 2024 2:07:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 02, 2024 2:07:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 02, 2024 2:07:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240502 14:07:47.848558 28345 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 02 14:07:47 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240502 14:07:48.000931 28345 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240502 14:07:48.020408 28345 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240502 14:07:48.853145 28345 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240502 14:07:48.853443 28345 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms
I20240502 14:07:48.853602 28345 mini_ranger_kms.cc:227] Using postgres at 127.27.174.84:52637
2024-05-02 14:07:50,189 [I] DB FLAVOR :POSTGRES
2024-05-02 14:07:50,190 [I] --------- Verifying Ranger DB connection ---------
2024-05-02 14:07:50,190 [I] Checking connection
2024-05-02 14:07:50,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-02 14:07:50,413 [I] connection success
2024-05-02 14:07:50,413 [I] --------- Verifying Ranger DB tables ---------
2024-05-02 14:07:50,413 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-02 14:07:50,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-02 14:07:50,655 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-02 14:07:50,655 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-02 14:07:50,655 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-02 14:07:50,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:52637/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-02 14:07:50,905 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240502 14:07:51.367683 28345 mini_ranger_kms.cc:310] Created kms service
I20240502 14:07:51.554155 28345 mini_ranger_kms.cc:326] Created kudu user
I20240502 14:07:51.595577 28345 mini_ranger_kms.cc:343] Created rangerkms user
May 02 14:07:51 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:07:52.002542 28345 mini_ranger_kms.cc:384] Added ranger policy
I20240502 14:07:52.003046 28345 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240502 14:07:52.003166 28345 mini_ranger_kms.cc:240] Using host: 127.27.174.84
I20240502 14:07:52.016209 28345 mini_ranger_kms.cc:276] Ranger KMS PID: 5749
I20240502 14:07:52.016616 28345 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.174.84:36621
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp] .....
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp].
May 02, 2024 2:07:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240502 14:07:54.090416 28345 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.090s user 0.307s sys 0.639s
I20240502 14:07:54.090885 28345 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240502 14:07:54.091281 28345 mini_ranger_kms.cc:401] 127.27.174.84:36621/kms/v1/keys
May 02 14:07:54 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714658874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 02 14:07:54 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:07:55 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658874, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:07:57 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:07:57.159122 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:32993
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:32993
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36621/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:07:57.442924 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:07:57.443467 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:07:57.443986 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:07:57.470958 5800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:07:57.471277 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:07:57.471489 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:07:57.471665 5800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:07:57.471838 5800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:07:57.503998 5800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.174.84:36621/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:32993
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:07:57.505065 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:07:57.505992 5800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:07:57.520205 5806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:07:57.520865 5809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:57.522367 5800 server_base.cc:1034] running on GCE node
W20240502 14:07:57.521389 5807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:07:58.558383 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:07:58.560789 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:07:58.562156 5800 hybrid_clock.cc:648] HybridClock initialized: now 1714658878562132 us; error 42 us; skew 500 ppm
I20240502 14:07:58.562918 5800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:07:58 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658878, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:07:58.572969 5800 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:07:58.577750 5800 webserver.cc:460] Webserver started at http://127.27.174.126:41831/ using document root <none> and password file <none>
I20240502 14:07:58.578646 5800 fs_manager.cc:359] Metadata directory not provided
I20240502 14:07:58.578830 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:07:58.579239 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:07:58 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658878, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:07:58.632503 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "fe86fcf2299340948c5ab5fc3b131b90"
format_stamp: "Formatted at 2024-05-02 14:07:58 on dist-test-slave-dlb8"
server_key: "0ef903b0152f777f347455e7b046c7ff"
server_key_iv: "364159d8d90d3a3ac6642a8ffe09fdba"
server_key_version: "kuduclusterkey@0"
I20240502 14:07:58.633940 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "fe86fcf2299340948c5ab5fc3b131b90"
format_stamp: "Formatted at 2024-05-02 14:07:58 on dist-test-slave-dlb8"
server_key: "0ef903b0152f777f347455e7b046c7ff"
server_key_iv: "364159d8d90d3a3ac6642a8ffe09fdba"
server_key_version: "kuduclusterkey@0"
I20240502 14:07:58.641757 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240502 14:07:58.661821 5818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:07:58.662952 5800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:07:58.663272 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "fe86fcf2299340948c5ab5fc3b131b90"
format_stamp: "Formatted at 2024-05-02 14:07:58 on dist-test-slave-dlb8"
server_key: "0ef903b0152f777f347455e7b046c7ff"
server_key_iv: "364159d8d90d3a3ac6642a8ffe09fdba"
server_key_version: "kuduclusterkey@0"
I20240502 14:07:58.663617 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:07:58.723668 5800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:07:58.728694 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:07:58.729179 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:07:58.794191 5800 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:32993
I20240502 14:07:58.794274 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:32993 every 8 connection(s)
I20240502 14:07:58.796772 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:07:58.801496 5870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:07:58.803025 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 5800
I20240502 14:07:58.803678 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:07:58.804826 28345 external_mini_cluster.cc:1428] Setting key 24d3299a3f055d551e5e7fcd9a6cedd5
I20240502 14:07:58.821482 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: Bootstrap starting.
May 02 14:07:58 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:07:58.828044 5870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: Neither blocks nor log segments found. Creating new log.
I20240502 14:07:58.829537 5870 log.cc:824] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: Log is configured to *not* fsync() on all Append() calls
I20240502 14:07:58.834892 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: No bootstrap required, opened a new log
I20240502 14:07:58.838635 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:07:58.807799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.27.174.126:32993)
0502 14:07:58.810445 (+ 2646us) server_negotiation.cc:206] Beginning negotiation
0502 14:07:58.810469 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:07:58.810523 (+ 54us) server_negotiation.cc:407] Connection header received
0502 14:07:58.814170 (+ 3647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:07:58.814208 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:07:58.814531 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:07:58.814814 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:07:58.818476 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:58.819768 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:07:58.821389 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:07:58.821830 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:07:58.828006 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:07:58.828096 (+ 90us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:07:58.828121 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:07:58.828213 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:07:58.832695 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:58.833893 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:58.833917 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:58.833952 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:58.834155 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:07:58.834731 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:07:58.834748 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:07:58.834760 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:07:58.835309 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:07:58.835612 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0502 14:07:58.836298 (+ 686us) server_negotiation.cc:299] Negotiation successful
0502 14:07:58.836699 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2134,"thread_start_us":1678,"threads_started":1}
I20240502 14:07:58.854617 5870 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } }
I20240502 14:07:58.855258 5870 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:07:58.855463 5870 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe86fcf2299340948c5ab5fc3b131b90, State: Initialized, Role: FOLLOWER
I20240502 14:07:58.856060 5870 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } }
I20240502 14:07:58.856570 5870 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:07:58.856817 5870 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:07:58.857101 5870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:07:58.861166 5870 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } }
I20240502 14:07:58.861802 5870 leader_election.cc:302] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe86fcf2299340948c5ab5fc3b131b90; no voters:
I20240502 14:07:58.863402 5870 leader_election.cc:288] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:07:58.863746 5875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:07:58.866979 5875 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [term 1 LEADER]: Becoming Leader. State: Replica: fe86fcf2299340948c5ab5fc3b131b90, State: Running, Role: LEADER
I20240502 14:07:58.867931 5870 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:07:58.868232 5875 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } }
I20240502 14:07:58.878006 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe86fcf2299340948c5ab5fc3b131b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } } }
I20240502 14:07:58.878854 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:58.880999 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe86fcf2299340948c5ab5fc3b131b90. Latest consensus state: current_term: 1 leader_uuid: "fe86fcf2299340948c5ab5fc3b131b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe86fcf2299340948c5ab5fc3b131b90" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 32993 } } }
I20240502 14:07:58.881912 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90 [sys.catalog]: This master's current role is: LEADER
I20240502 14:07:58.887661 5885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:07:58.897073 5885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:07:58.898093 5870 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240502 14:07:58.927084 5885 catalog_manager.cc:1347] Generated new cluster ID: bc75495422684f1f955e2a66d60ca250
I20240502 14:07:58.927503 5885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:07:58.946190 5885 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:07:58.948457 5885 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:07:58.965880 5885 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: Generated new TSK 0
I20240502 14:07:58.967100 5885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:07:59.677574 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:00.007490 5866 debug-util.cc:398] Leaking SignalData structure 0x7b0800075060 after lost signal to thread 5801
W20240502 14:08:00.008244 5866 debug-util.cc:398] Leaking SignalData structure 0x7b08000757e0 after lost signal to thread 5869
W20240502 14:08:00.008637 5866 debug-util.cc:398] Leaking SignalData structure 0x7b0800075ac0 after lost signal to thread 5894
May 02 14:08:00 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714658880, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240502 14:08:00.120671 5915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:00.121456 5915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:00.130769 5915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:00.228158 5915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:00.228753 5915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:00.230415 5915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:08:00.299271 5915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.174.84:36621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:00.301000 5915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:00.302443 5915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:00.345204 5926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
May 02 14:08:00 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658880, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:08:01 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658880, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
W20240502 14:08:00.350013 5929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:00.354048 5927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:00.354365 5915 server_base.cc:1034] running on GCE node
I20240502 14:08:01.906332 5915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:01.918256 5915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:01.919708 5915 hybrid_clock.cc:648] HybridClock initialized: now 1714658881919638 us; error 87 us; skew 500 ppm
I20240502 14:08:01.920781 5915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:01 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658881, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:01.928416 5915 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:08:01.956450 5915 webserver.cc:460] Webserver started at http://127.27.174.65:33565/ using document root <none> and password file <none>
I20240502 14:08:01.957862 5915 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:01.958266 5915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:01.958911 5915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:01 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658881, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:01.994860 5915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "138e9f75bfa9462f8f2b3eab77402fa1"
format_stamp: "Formatted at 2024-05-02 14:08:01 on dist-test-slave-dlb8"
server_key: "d8358ccf2ab7042995734edc34a55112"
server_key_iv: "c4977fe25c8dacdcfcfc015c46393877"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:02.011310 5915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "138e9f75bfa9462f8f2b3eab77402fa1"
format_stamp: "Formatted at 2024-05-02 14:08:01 on dist-test-slave-dlb8"
server_key: "d8358ccf2ab7042995734edc34a55112"
server_key_iv: "c4977fe25c8dacdcfcfc015c46393877"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:02.033701 5915 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.009s sys 0.004s
I20240502 14:08:02.076768 5939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:02.078243 5915 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.004s
I20240502 14:08:02.078648 5915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "138e9f75bfa9462f8f2b3eab77402fa1"
format_stamp: "Formatted at 2024-05-02 14:08:01 on dist-test-slave-dlb8"
server_key: "d8358ccf2ab7042995734edc34a55112"
server_key_iv: "c4977fe25c8dacdcfcfc015c46393877"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:02.079178 5915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:02.180395 5915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:02.188112 5915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:02.188721 5915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:02.198985 5915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:02.210525 5915 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:02.210872 5915 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:08:02.211244 5915 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:02.211514 5915 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:02.621299 5915 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:38685
I20240502 14:08:02.621418 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:38685 every 8 connection(s)
I20240502 14:08:02.625026 5915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:08:02.630995 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 5915
I20240502 14:08:02.631742 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:08:02.633682 28345 external_mini_cluster.cc:1428] Setting key f21fa6e5009d2e03bf5964f61e8f7b38
May 02 14:08:02 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658881, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:08:02.699831 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:02.643003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:56859 (local address 127.27.174.126:32993)
0502 14:08:02.663300 (+ 20297us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:02.663341 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:02.663402 (+ 61us) server_negotiation.cc:407] Connection header received
0502 14:08:02.669553 (+ 6151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:02.669590 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:02.669873 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:02.670083 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:02.673666 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:02.675205 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:02.676553 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:02.677106 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:02.682266 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:02.682380 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:02.682426 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:02.682530 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:02.689538 (+ 7008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:02.693123 (+ 3585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:02.693170 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:02.693217 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:02.693442 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:02.694352 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:02.694395 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:02.694420 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:02.694919 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:02.695187 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:02.697091 (+ 1904us) server_negotiation.cc:299] Negotiation successful
0502 14:08:02.697608 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19662,"thread_start_us":19282,"threads_started":1}
I20240502 14:08:02.702419 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:02.640540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:32993 (local address 127.27.174.65:56859)
0502 14:08:02.659477 (+ 18937us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:02.659587 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:02.667575 (+ 7988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:02.670729 (+ 3154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:02.670774 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:02.671832 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:02.673273 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:02.673344 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:02.675443 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:02.675464 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:02.676306 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:02.676340 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:02.676715 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:02.677797 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:02.677913 (+ 116us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:02.681954 (+ 4041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:02.691622 (+ 9668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:02.691656 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:02.691706 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:02.692741 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:02.693693 (+ 952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:02.693730 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:02.693761 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:02.694068 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:02.695336 (+ 1268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:02.695378 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:02.696201 (+ 823us) client_negotiation.cc:769] Sending connection context
0502 14:08:02.698872 (+ 2671us) client_negotiation.cc:240] Negotiation successful
0502 14:08:02.699454 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18147,"thread_start_us":17645,"threads_started":1}
I20240502 14:08:02.704685 6059 heartbeater.cc:347] Connected to a master server at 127.27.174.126:32993
I20240502 14:08:02.705127 6059 heartbeater.cc:464] Registering TS with master...
I20240502 14:08:02.706430 6059 heartbeater.cc:511] Master 127.27.174.126:32993 requested a full tablet report, sending...
I20240502 14:08:02.709578 5835 ts_manager.cc:194] Registered new tserver with Master: 138e9f75bfa9462f8f2b3eab77402fa1 (127.27.174.65:38685)
I20240502 14:08:02.712563 5835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:56859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:08:02.728773 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:03.025548 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:03.026007 6065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:03.026618 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:03.054571 6065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:03.054898 6065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:03.055727 6065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:08:03.088306 6065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.174.84:36621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:03.089357 6065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:03.090327 6065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:03.108258 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:03.716570 6059 heartbeater.cc:503] Master 127.27.174.126:32993 was elected leader, sending a full tablet report...
W20240502 14:08:03.108287 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:03.110757 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:04.136886 6073 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:08:04.136978 6065 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:08:04.142769 6065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:04.145089 6065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:04.146432 6065 hybrid_clock.cc:648] HybridClock initialized: now 1714658884146397 us; error 43 us; skew 500 ppm
I20240502 14:08:04.147221 6065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:04 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:04.158325 6065 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:08:04.162518 6065 webserver.cc:460] Webserver started at http://127.27.174.66:45385/ using document root <none> and password file <none>
I20240502 14:08:04.163354 6065 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:04.163555 6065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:04.163946 6065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:04 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:04.181412 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "5e1b220cf51645dbbc3aa75f95352a2d"
format_stamp: "Formatted at 2024-05-02 14:08:04 on dist-test-slave-dlb8"
server_key: "518780dad660c9d890faa97ea2994170"
server_key_iv: "882793d11b923b36985df8803f16f943"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:04.182677 6065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "5e1b220cf51645dbbc3aa75f95352a2d"
format_stamp: "Formatted at 2024-05-02 14:08:04 on dist-test-slave-dlb8"
server_key: "518780dad660c9d890faa97ea2994170"
server_key_iv: "882793d11b923b36985df8803f16f943"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:04.189767 6065 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:08:04.210178 6082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:04.211211 6065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:08:04.211527 6065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "5e1b220cf51645dbbc3aa75f95352a2d"
format_stamp: "Formatted at 2024-05-02 14:08:04 on dist-test-slave-dlb8"
server_key: "518780dad660c9d890faa97ea2994170"
server_key_iv: "882793d11b923b36985df8803f16f943"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:04.211874 6065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:04.261184 6065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:04.266336 6065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:04.266778 6065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:04.269184 6065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:04.272836 6065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:04.273032 6065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:04.273275 6065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:04.273425 6065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:04.399919 6065 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:43279
I20240502 14:08:04.400004 6194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:43279 every 8 connection(s)
I20240502 14:08:04.402257 6065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:08:04.406740 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 6065
I20240502 14:08:04.407407 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:08:04.408916 28345 external_mini_cluster.cc:1428] Setting key 7badaaf0fc4ae3f2bad0835488b36b5a
May 02 14:08:04 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:08:04.435024 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:04.405907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58697 (local address 127.27.174.126:32993)
0502 14:08:04.410556 (+ 4649us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:04.410571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:04.410608 (+ 37us) server_negotiation.cc:407] Connection header received
0502 14:08:04.415217 (+ 4609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:04.415251 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:04.415504 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:04.415698 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:04.417783 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:04.419236 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:04.420783 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:04.421339 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:04.425442 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:04.425517 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:04.425534 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:04.425620 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:04.429315 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:04.430330 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:04.430351 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:04.430364 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:04.430500 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:04.431040 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:04.431059 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:04.431075 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:04.431456 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:04.431665 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:04.432530 (+ 865us) server_negotiation.cc:299] Negotiation successful
0502 14:08:04.432936 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4119,"thread_start_us":3834,"threads_started":1}
I20240502 14:08:04.435541 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:04.406240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:32993 (local address 127.27.174.66:58697)
0502 14:08:04.409725 (+ 3485us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:04.409830 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:04.414858 (+ 5028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:04.415885 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:04.415908 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:04.416552 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:04.417496 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:04.417540 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:04.419618 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:04.419643 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:04.420501 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:04.420549 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:04.420850 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:04.421703 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:04.421808 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:04.425126 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:04.429511 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:04.429539 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:04.429561 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:04.430164 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:04.430668 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:04.430685 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:04.430697 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:04.430867 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:04.431636 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:04.431669 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:04.432014 (+ 345us) client_negotiation.cc:769] Sending connection context
0502 14:08:04.432319 (+ 305us) client_negotiation.cc:240] Negotiation successful
0502 14:08:04.432838 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":1618,"threads_started":1}
I20240502 14:08:04.437407 6195 heartbeater.cc:347] Connected to a master server at 127.27.174.126:32993
I20240502 14:08:04.437776 6195 heartbeater.cc:464] Registering TS with master...
I20240502 14:08:04.438776 6195 heartbeater.cc:511] Master 127.27.174.126:32993 requested a full tablet report, sending...
I20240502 14:08:04.440785 5835 ts_manager.cc:194] Registered new tserver with Master: 5e1b220cf51645dbbc3aa75f95352a2d (127.27.174.66:43279)
I20240502 14:08:04.442065 5835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:58697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:08:04.459050 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:04.748855 6201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:04.749289 6201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:04.749840 6201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:04.778875 6201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:04.779152 6201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:04.779920 6201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:08:04.812181 6201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.174.84:36621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:32993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:04.813141 6201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:04.814024 6201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:04.828685 6208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:05.444648 6195 heartbeater.cc:503] Master 127.27.174.126:32993 was elected leader, sending a full tablet report...
W20240502 14:08:04.829555 6207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:04.830054 6210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:04.831815 6201 server_base.cc:1034] running on GCE node
I20240502 14:08:05.850283 6201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:05.852437 6201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:05.853768 6201 hybrid_clock.cc:648] HybridClock initialized: now 1714658885853740 us; error 44 us; skew 500 ppm
I20240502 14:08:05.854502 6201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:05 dist-test-slave-dlb8 krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658885, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:05.860028 6201 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:08:05.865238 6201 webserver.cc:460] Webserver started at http://127.27.174.67:35359/ using document root <none> and password file <none>
I20240502 14:08:05.866130 6201 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:05.866346 6201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:05.866747 6201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:05 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658885, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:05.885550 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "79f2438a7aae4789ad8d8733b0d69743"
format_stamp: "Formatted at 2024-05-02 14:08:05 on dist-test-slave-dlb8"
server_key: "bf4492e048bab4fb57407d8193711ce6"
server_key_iv: "168588c6b90df345f6cad913c56ce667"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:05.886600 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "79f2438a7aae4789ad8d8733b0d69743"
format_stamp: "Formatted at 2024-05-02 14:08:05 on dist-test-slave-dlb8"
server_key: "bf4492e048bab4fb57407d8193711ce6"
server_key_iv: "168588c6b90df345f6cad913c56ce667"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:05.893534 6201 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:08:05.911401 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:05.912473 6201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20240502 14:08:05.912766 6201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "79f2438a7aae4789ad8d8733b0d69743"
format_stamp: "Formatted at 2024-05-02 14:08:05 on dist-test-slave-dlb8"
server_key: "bf4492e048bab4fb57407d8193711ce6"
server_key_iv: "168588c6b90df345f6cad913c56ce667"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:05.913091 6201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:05.971107 6201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:05.975656 6201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:05.976047 6201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:05.978473 6201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:05.982141 6201 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:05.982324 6201 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:05.982566 6201 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:05.982765 6201 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:06.106256 6201 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:35525
I20240502 14:08:06.106357 6330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:35525 every 8 connection(s)
I20240502 14:08:06.108657 6201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:08:06.109740 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 6201
I20240502 14:08:06.110373 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:08:06.111956 28345 external_mini_cluster.cc:1428] Setting key 956eb8ca62909ed17d6a57abb95b36cc
May 02 14:08:06 dist-test-slave-dlb8 krb5kdc[5352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658885, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:08:06.143345 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.113717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:32993 (local address 127.27.174.67:57313)
0502 14:08:06.116819 (+ 3102us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.116889 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.121540 (+ 4651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.122712 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.122737 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.123569 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:06.124730 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.124773 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.126103 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.126142 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.126945 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.126978 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.127351 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.128389 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:06.128488 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:06.131826 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:06.137258 (+ 5432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:06.137281 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:06.137307 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:06.137916 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:06.138548 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:06.138566 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:06.138579 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:06.138755 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:06.139599 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:06.139622 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:06.139968 (+ 346us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.140266 (+ 298us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.140708 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":1801,"threads_started":1}
I20240502 14:08:06.143585 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.113474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:57313 (local address 127.27.174.126:32993)
0502 14:08:06.115120 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.115147 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.117031 (+ 1884us) server_negotiation.cc:407] Connection header received
0502 14:08:06.121931 (+ 4900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.121953 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.122183 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.122333 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:06.124975 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.125833 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.127180 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.127752 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.132084 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:06.132153 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:06.132172 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:06.132271 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:06.136968 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:06.138120 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:06.138157 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:06.138179 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:06.138371 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:06.138912 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:06.138936 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:06.138957 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:06.139413 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:06.139655 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.140472 (+ 817us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.140935 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":556,"threads_started":1}
I20240502 14:08:06.146270 6331 heartbeater.cc:347] Connected to a master server at 127.27.174.126:32993
I20240502 14:08:06.146677 6331 heartbeater.cc:464] Registering TS with master...
I20240502 14:08:06.147922 6331 heartbeater.cc:511] Master 127.27.174.126:32993 requested a full tablet report, sending...
I20240502 14:08:06.150115 5834 ts_manager.cc:194] Registered new tserver with Master: 79f2438a7aae4789ad8d8733b0d69743 (127.27.174.67:35525)
I20240502 14:08:06.151468 5834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:57313
I20240502 14:08:06.162165 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:08:06.198920 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.178684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.27.174.126:32993)
0502 14:08:06.179451 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.179474 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.180302 (+ 828us) server_negotiation.cc:407] Connection header received
0502 14:08:06.180634 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.180650 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.180801 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.180925 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:06.184598 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.185381 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.187186 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.187540 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.190819 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:06.190887 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:06.190903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:06.190985 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:06.194008 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:06.195081 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:06.195100 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:06.195116 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:06.195249 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:06.195695 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:06.195711 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:06.195723 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:06.196127 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:06.196398 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.196891 (+ 493us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.197388 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20240502 14:08:06.205476 5834 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:08:06.226457 5834 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:08:06.268630 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.251167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35525 (local address 127.0.0.1:53872)
0502 14:08:06.255654 (+ 4487us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.255716 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.256010 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.257066 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.257107 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.257266 (+ 159us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:08:06.258312 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.258366 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.262974 (+ 4608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.262994 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.264925 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.265007 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.265371 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.265496 (+ 125us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.265752 (+ 256us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.266080 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3878,"thread_start_us":7557,"threads_started":1}
I20240502 14:08:06.269738 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.251434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.27.174.67:35525)
0502 14:08:06.254393 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.254418 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.255867 (+ 1449us) server_negotiation.cc:407] Connection header received
0502 14:08:06.256338 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.256371 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.256763 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.256999 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:08:06.259004 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.261871 (+ 2867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.265924 (+ 4053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.267337 (+ 1413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.267592 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.267799 (+ 207us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.268031 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2312,"thread_start_us":2237,"threads_started":1}
I20240502 14:08:06.274256 6266 tablet_service.cc:1467] Processing CreateTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd (DEFAULT_TABLE table=test-table [id=837b1926ead5445a9c47ea21012bbaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:08:06.277246 6266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68e0d1756ed4c1e9da552aef26d3efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:08:06.277499 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.252596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:43279 (local address 127.0.0.1:39296)
0502 14:08:06.259888 (+ 7292us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.260105 (+ 217us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.260340 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.263445 (+ 3105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.263472 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.263594 (+ 122us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:08:06.264673 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.264724 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.272591 (+ 7867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.272633 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.274450 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.274499 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.274894 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.275020 (+ 126us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.275198 (+ 178us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.275471 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6367,"thread_start_us":3513,"threads_started":1}
I20240502 14:08:06.278254 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.255241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.27.174.66:43279)
0502 14:08:06.262458 (+ 7217us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.262490 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.262554 (+ 64us) server_negotiation.cc:407] Connection header received
0502 14:08:06.262739 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.262774 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.263168 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.263447 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:08:06.269779 (+ 6332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.272366 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.274706 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.275889 (+ 1183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.276120 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.276357 (+ 237us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.276597 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6306,"thread_start_us":12316,"threads_started":1}
I20240502 14:08:06.279700 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.259659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38685 (local address 127.0.0.1:49056)
0502 14:08:06.268041 (+ 8382us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.268167 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.268416 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.270694 (+ 2278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.270722 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.270815 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:08:06.272049 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.272127 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.274914 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.274944 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.276643 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.276698 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.276993 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.277071 (+ 78us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.277249 (+ 178us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.277451 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7430,"thread_start_us":6813,"threads_started":1}
I20240502 14:08:06.282011 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.261471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.27.174.65:38685)
0502 14:08:06.269353 (+ 7882us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.269380 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.269434 (+ 54us) server_negotiation.cc:407] Connection header received
0502 14:08:06.269617 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.269647 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.270003 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.270229 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:08:06.272420 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.274624 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.278429 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.279732 (+ 1303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.279995 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.280202 (+ 207us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.280422 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7254,"thread_start_us":7512,"threads_started":1}
I20240502 14:08:06.282577 6130 tablet_service.cc:1467] Processing CreateTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd (DEFAULT_TABLE table=test-table [id=837b1926ead5445a9c47ea21012bbaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:08:06.284502 6130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68e0d1756ed4c1e9da552aef26d3efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:08:06.286358 5994 tablet_service.cc:1467] Processing CreateTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd (DEFAULT_TABLE table=test-table [id=837b1926ead5445a9c47ea21012bbaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:08:06.288142 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68e0d1756ed4c1e9da552aef26d3efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:08:06.308823 6350 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Bootstrap starting.
I20240502 14:08:06.312070 6351 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Bootstrap starting.
I20240502 14:08:06.316969 6350 tablet_bootstrap.cc:654] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Neither blocks nor log segments found. Creating new log.
I20240502 14:08:06.319464 6351 tablet_bootstrap.cc:654] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Neither blocks nor log segments found. Creating new log.
I20240502 14:08:06.321193 6352 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Bootstrap starting.
I20240502 14:08:06.321827 6351 log.cc:824] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Log is configured to *not* fsync() on all Append() calls
I20240502 14:08:06.321957 6350 log.cc:824] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:08:06.330807 6352 tablet_bootstrap.cc:654] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Neither blocks nor log segments found. Creating new log.
I20240502 14:08:06.333042 6351 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: No bootstrap required, opened a new log
I20240502 14:08:06.333051 6352 log.cc:824] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:08:06.333463 6351 ts_tablet_manager.cc:1367] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.015s
I20240502 14:08:06.334302 6350 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: No bootstrap required, opened a new log
I20240502 14:08:06.334806 6350 ts_tablet_manager.cc:1367] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.003s
I20240502 14:08:06.339411 6352 tablet_bootstrap.cc:492] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: No bootstrap required, opened a new log
I20240502 14:08:06.339880 6352 ts_tablet_manager.cc:1367] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20240502 14:08:06.354643 6351 raft_consensus.cc:348] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.355317 6351 raft_consensus.cc:374] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:08:06.355587 6351 raft_consensus.cc:729] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f2438a7aae4789ad8d8733b0d69743, State: Initialized, Role: FOLLOWER
I20240502 14:08:06.356292 6351 consensus_queue.cc:260] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.359277 6331 heartbeater.cc:503] Master 127.27.174.126:32993 was elected leader, sending a full tablet report...
I20240502 14:08:06.360086 6351 ts_tablet_manager.cc:1398] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Time spent starting tablet: real 0.026s user 0.020s sys 0.008s
W20240502 14:08:06.364127 6332 tablet.cc:2369] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:08:06.364589 6350 raft_consensus.cc:348] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.365453 6350 raft_consensus.cc:374] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:08:06.365741 6350 raft_consensus.cc:729] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1b220cf51645dbbc3aa75f95352a2d, State: Initialized, Role: FOLLOWER
I20240502 14:08:06.366569 6350 consensus_queue.cc:260] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.367995 6352 raft_consensus.cc:348] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.368832 6352 raft_consensus.cc:374] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:08:06.369091 6352 raft_consensus.cc:729] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138e9f75bfa9462f8f2b3eab77402fa1, State: Initialized, Role: FOLLOWER
I20240502 14:08:06.369897 6352 consensus_queue.cc:260] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.371176 6350 ts_tablet_manager.cc:1398] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Time spent starting tablet: real 0.036s user 0.031s sys 0.005s
I20240502 14:08:06.373858 6352 ts_tablet_manager.cc:1398] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Time spent starting tablet: real 0.034s user 0.024s sys 0.008s
W20240502 14:08:06.408167 6196 tablet.cc:2369] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:08:06.422835 6060 tablet.cc:2369] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:08:06.514453 6357 raft_consensus.cc:482] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:06.514926 6357 raft_consensus.cc:504] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.517199 6357 leader_election.cc:288] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79f2438a7aae4789ad8d8733b0d69743 (127.27.174.67:35525), 138e9f75bfa9462f8f2b3eab77402fa1 (127.27.174.65:38685)
I20240502 14:08:06.527704 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.517782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57943 (local address 127.27.174.67:35525)
0502 14:08:06.518645 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.518694 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.520433 (+ 1739us) server_negotiation.cc:407] Connection header received
0502 14:08:06.520848 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.520876 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.521120 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.521249 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:08:06.522718 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.523772 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.525707 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.526596 (+ 889us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.526684 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.526797 (+ 113us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.526917 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20240502 14:08:06.527921 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.517646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:35525 (local address 127.27.174.66:57943)
0502 14:08:06.520278 (+ 2632us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.520328 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.520558 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.521456 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.521484 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.521574 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:08:06.522470 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.522509 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.524018 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.524038 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.525457 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.525502 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.525838 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.525932 (+ 94us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.526085 (+ 153us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.526272 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"thread_start_us":1701,"threads_started":1}
I20240502 14:08:06.529610 6286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68e0d1756ed4c1e9da552aef26d3efd" candidate_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f2438a7aae4789ad8d8733b0d69743" is_pre_election: true
I20240502 14:08:06.530017 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.520170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38685 (local address 127.27.174.66:54977)
0502 14:08:06.522827 (+ 2657us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.522860 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:06.523050 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:06.523713 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:06.523731 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:06.523795 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:08:06.524629 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.524663 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.526308 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.526347 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:06.528115 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:06.528166 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.528503 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.528606 (+ 103us) client_negotiation.cc:769] Sending connection context
0502 14:08:06.528735 (+ 129us) client_negotiation.cc:240] Negotiation successful
0502 14:08:06.528940 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"thread_start_us":1844,"threads_started":1}
I20240502 14:08:06.530462 6286 raft_consensus.cc:2454] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1b220cf51645dbbc3aa75f95352a2d in term 0.
I20240502 14:08:06.530993 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.520477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:54977 (local address 127.27.174.65:38685)
0502 14:08:06.521207 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.521244 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.522962 (+ 1718us) server_negotiation.cc:407] Connection header received
0502 14:08:06.523294 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.523319 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.523537 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.523685 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:08:06.524856 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.525981 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.528412 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.529284 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.529421 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.529613 (+ 192us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.529806 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20240502 14:08:06.531579 6083 leader_election.cc:302] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1b220cf51645dbbc3aa75f95352a2d, 79f2438a7aae4789ad8d8733b0d69743; no voters:
I20240502 14:08:06.532260 6357 raft_consensus.cc:2781] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:08:06.532542 6357 raft_consensus.cc:482] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:08:06.532307 6014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68e0d1756ed4c1e9da552aef26d3efd" candidate_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" is_pre_election: true
I20240502 14:08:06.532773 6357 raft_consensus.cc:3037] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:08:06.533026 6014 raft_consensus.cc:2454] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1b220cf51645dbbc3aa75f95352a2d in term 0.
I20240502 14:08:06.537374 6357 raft_consensus.cc:504] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.538786 6357 leader_election.cc:288] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [CANDIDATE]: Term 1 election: Requested vote from peers 79f2438a7aae4789ad8d8733b0d69743 (127.27.174.67:35525), 138e9f75bfa9462f8f2b3eab77402fa1 (127.27.174.65:38685)
I20240502 14:08:06.539445 6286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68e0d1756ed4c1e9da552aef26d3efd" candidate_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f2438a7aae4789ad8d8733b0d69743"
I20240502 14:08:06.539731 6014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68e0d1756ed4c1e9da552aef26d3efd" candidate_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138e9f75bfa9462f8f2b3eab77402fa1"
I20240502 14:08:06.539877 6286 raft_consensus.cc:3037] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:08:06.540141 6014 raft_consensus.cc:3037] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:08:06.544370 6286 raft_consensus.cc:2454] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1b220cf51645dbbc3aa75f95352a2d in term 1.
I20240502 14:08:06.544839 6014 raft_consensus.cc:2454] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1b220cf51645dbbc3aa75f95352a2d in term 1.
I20240502 14:08:06.545092 6083 leader_election.cc:302] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1b220cf51645dbbc3aa75f95352a2d, 79f2438a7aae4789ad8d8733b0d69743; no voters:
I20240502 14:08:06.545692 6357 raft_consensus.cc:2781] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:08:06.547775 6357 raft_consensus.cc:686] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 1 LEADER]: Becoming Leader. State: Replica: 5e1b220cf51645dbbc3aa75f95352a2d, State: Running, Role: LEADER
I20240502 14:08:06.548605 6357 consensus_queue.cc:237] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } }
I20240502 14:08:06.559861 5834 catalog_manager.cc:5484] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1b220cf51645dbbc3aa75f95352a2d (127.27.174.66). New cstate: current_term: 1 leader_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1b220cf51645dbbc3aa75f95352a2d" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 43279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20240502 14:08:06.588744 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.582399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.27.174.66:43279)
0502 14:08:06.582871 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:06.582890 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:06.582960 (+ 70us) server_negotiation.cc:407] Connection header received
0502 14:08:06.583324 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:06.583343 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:06.583590 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:06.583726 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:08:06.584390 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.585144 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:06.586358 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:06.586696 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:06.586866 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:08:06.587359 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:08:06.587499 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:06.587682 (+ 183us) server_negotiation.cc:299] Negotiation successful
0502 14:08:06.587794 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20240502 14:08:06.601025 6014 raft_consensus.cc:1261] T f68e0d1756ed4c1e9da552aef26d3efd P 138e9f75bfa9462f8f2b3eab77402fa1 [term 1 FOLLOWER]: Refusing update from remote peer 5e1b220cf51645dbbc3aa75f95352a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:08:06.601092 6286 raft_consensus.cc:1261] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 1 FOLLOWER]: Refusing update from remote peer 5e1b220cf51645dbbc3aa75f95352a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:08:06.602494 6361 consensus_queue.cc:1035] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f2438a7aae4789ad8d8733b0d69743" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:08:06.603107 6357 consensus_queue.cc:1035] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "138e9f75bfa9462f8f2b3eab77402fa1" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:08:06.660306 5834 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:08:06.660890 5834 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55378:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:08:06.672394 5834 catalog_manager.cc:5771] T 00000000000000000000000000000000 P fe86fcf2299340948c5ab5fc3b131b90: Sending DeleteTablet for 3 replicas of tablet f68e0d1756ed4c1e9da552aef26d3efd
I20240502 14:08:06.674734 6130 tablet_service.cc:1514] Processing DeleteTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:08:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:39296
I20240502 14:08:06.675127 6266 tablet_service.cc:1514] Processing DeleteTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:08:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:53872
I20240502 14:08:06.678901 6373 tablet_replica.cc:330] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: stopping tablet replica
I20240502 14:08:06.679704 5994 tablet_service.cc:1514] Processing DeleteTablet for tablet f68e0d1756ed4c1e9da552aef26d3efd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:08:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:49056
I20240502 14:08:06.680608 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 5915
I20240502 14:08:06.684921 6374 tablet_replica.cc:330] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: stopping tablet replica
I20240502 14:08:06.686975 6373 raft_consensus.cc:2229] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:08:06.687647 6374 raft_consensus.cc:2229] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:08:06.689076 6374 raft_consensus.cc:2258] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:08:06.690003 6373 raft_consensus.cc:2258] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:08:06.694288 6374 ts_tablet_manager.cc:1875] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:08:06.695948 6373 ts_tablet_manager.cc:1875] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:08:06.711913 6374 ts_tablet_manager.cc:1888] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:08:06.712002 6373 ts_tablet_manager.cc:1888] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:08:06.712347 6373 log.cc:1196] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/f68e0d1756ed4c1e9da552aef26d3efd
I20240502 14:08:06.712486 6374 log.cc:1196] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/f68e0d1756ed4c1e9da552aef26d3efd
I20240502 14:08:06.713071 6373 ts_tablet_manager.cc:1909] T f68e0d1756ed4c1e9da552aef26d3efd P 79f2438a7aae4789ad8d8733b0d69743: Deleting consensus metadata
I20240502 14:08:06.713447 6374 ts_tablet_manager.cc:1909] T f68e0d1756ed4c1e9da552aef26d3efd P 5e1b220cf51645dbbc3aa75f95352a2d: Deleting consensus metadata
I20240502 14:08:06.716655 5822 catalog_manager.cc:4830] TS 5e1b220cf51645dbbc3aa75f95352a2d (127.27.174.66:43279): tablet f68e0d1756ed4c1e9da552aef26d3efd (table test-table [id=837b1926ead5445a9c47ea21012bbaa3]) successfully deleted
I20240502 14:08:06.716948 5819 catalog_manager.cc:4830] TS 79f2438a7aae4789ad8d8733b0d69743 (127.27.174.67:35525): tablet f68e0d1756ed4c1e9da552aef26d3efd (table test-table [id=837b1926ead5445a9c47ea21012bbaa3]) successfully deleted
W20240502 14:08:06.717981 5819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:38685 (error 108)
I20240502 14:08:06.718364 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 6065
I20240502 14:08:06.722865 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:06.721732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:38685 (local address 127.0.0.1:49060)
0502 14:08:06.722325 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:06.722506 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20240502 14:08:06.723266 5819 catalog_manager.cc:4573] TS 138e9f75bfa9462f8f2b3eab77402fa1 (127.27.174.65:38685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f68e0d1756ed4c1e9da552aef26d3efd: Network error: Client connection negotiation failed: client connection to 127.27.174.65:38685: connect: Connection refused (error 111)
I20240502 14:08:06.746394 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 6201
I20240502 14:08:06.768718 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 5800
I20240502 14:08:06.800745 28345 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240502 14:08:07.147897 28345 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240502 14:08:07.148312 28345 mini_ranger.cc:66] Stopping Ranger...
I20240502 14:08:07.495394 28345 mini_ranger.cc:68] Stopped Ranger
2024-05-02 14:08:07.495 UTC [5383] LOG: received smart shutdown request
2024-05-02 14:08:07.500 UTC [5383] LOG: background worker "logical replication launcher" (PID 5392) exited with exit code 1
2024-05-02 14:08:07.500 UTC [5387] LOG: shutting down
2024-05-02 14:08:07.663 UTC [5383] LOG: database system is shut down
2024-05-02T14:08:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53250 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:08:07 dist-test-slave-dlb8 krb5kdc[6386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:08:07 dist-test-slave-dlb8 krb5kdc[6386](info): set up 2 sockets
May 02 14:08:07 dist-test-slave-dlb8 krb5kdc[6386](info): commencing operation
krb5kdc: starting...
W20240502 14:08:08.788636 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.010s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:08:08 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:08:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangeradmin_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangerlookup_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
I20240502 14:08:09.080789 28345 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/postgres -l logfile start
2024-05-02 14:08:11.454 UTC [6416] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-02 14:08:11.454 UTC [6416] LOG: listening on IPv4 address "127.27.174.84", port 35591
2024-05-02 14:08:11.457 UTC [6416] LOG: listening on Unix socket "/tmp/.s.PGSQL.35591"
2024-05-02 14:08:11.480 UTC [6419] LOG: database system was shut down at 2024-05-02 14:08:10 UTC
2024-05-02 14:08:11.483 UTC [6416] LOG: database system is ready to accept connections
I20240502 14:08:12.335896 28345 mini_postgres.cc:96] Postgres bound to 35591
2024-05-02 14:08:12.351 UTC [6427] FATAL: database "slave" does not exist
127.27.174.84:35591 - accepting connections
I20240502 14:08:12.352708 28345 mini_ranger.cc:161] Starting Ranger...
I20240502 14:08:12.376869 28345 mini_ranger.cc:84] Created miniranger Postgres user
I20240502 14:08:12.854287 28345 mini_ranger.cc:87] Created ranger Postgres database
I20240502 14:08:12.854514 28345 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin
2024-05-02 14:08:13,072 [I] DB FLAVOR :POSTGRES
2024-05-02 14:08:13,073 [I] --------- Verifying Ranger DB connection ---------
2024-05-02 14:08:13,073 [I] Checking connection..
2024-05-02 14:08:13,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-02 14:08:13,292 [I] Checking connection passed.
2024-05-02 14:08:13,292 [I] --------- Verifying version history table ---------
2024-05-02 14:08:13,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:08:13,534 [I] Table x_db_version_h does not exist in database ranger
2024-05-02 14:08:13,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:08:13,774 [I] Table x_db_version_h does not exist in database ranger
2024-05-02 14:08:13,774 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-02 14:08:13,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-02 14:08:14.000 UTC [6491] WARNING: there is no transaction in progress
2024-05-02 14:08:14,006 [I] create_dbversion_catalog.sql file imported successfully
2024-05-02 14:08:14,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-02 14:08:14,230 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-02 14:08:14,230 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-02 14:08:14,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-02 14:08:14,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-02 14:08:14,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dlb8.c.gcp-upstream.internal','N') ;"
2024-05-02 14:08:14,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-02 14:08:15,140 [I] Table x_portal_user does not exist in database ranger
2024-05-02 14:08:15,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-02 14:08:15,366 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-02 14:08:15,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-02 14:08:15,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-02 14:08:15,832 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-02 14:08:15,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-02 14:08:16.728 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.744 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.760 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.774 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.789 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.806 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.858 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.864 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.878 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.887 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.896 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.909 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.918 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.924 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.930 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:16.936 UTC [6599] WARNING: there is no transaction in progress
2024-05-02 14:08:17,527 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-02 14:08:17,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-02 14:08:17,750 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-02 14:08:17,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-02 14:08:17,984 [I] Table x_portal_user already exists in database 'ranger'
2024-05-02 14:08:17,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-02 14:08:18,205 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-02 14:08:18,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-02 14:08:18,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-02 14:08:18,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dlb8.c.gcp-upstream.internal';"
2024-05-02 14:08:18,876 [I] CORE_DB_SCHEMA import status has been updated
2024-05-02 14:08:18,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-02 14:08:19,103 [I] DB_PATCHES have already been applied
I20240502 14:08:19.108976 28345 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTLmxo8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240502 14:08:19.109253 28345 mini_ranger.cc:193] Using host: 127.27.174.84
I20240502 14:08:19.122202 28345 mini_ranger.cc:227] Ranger admin URL: http://127.27.174.84:49347
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp] .....
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-admin/ews/webapp].
May 02, 2024 2:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240502 14:08:20.105356 28345 mini_ranger.cc:160] Time spent starting Ranger: real 7.753s user 0.002s sys 0.039s
May 02, 2024 2:08:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 02, 2024 2:08:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
May 02, 2024 2:08:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 02, 2024 2:08:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 02, 2024 2:08:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240502 14:08:40.536396 28345 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/rangerkms_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/HTTP_127.27.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 02 14:08:40 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240502 14:08:40.713840 28345 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240502 14:08:40.733289 28345 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240502 14:08:41.566962 28345 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240502 14:08:41.567307 28345 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms
I20240502 14:08:41.567487 28345 mini_ranger_kms.cc:227] Using postgres at 127.27.174.84:35591
2024-05-02 14:08:42,419 [I] DB FLAVOR :POSTGRES
2024-05-02 14:08:42,420 [I] --------- Verifying Ranger DB connection ---------
2024-05-02 14:08:42,420 [I] Checking connection
2024-05-02 14:08:42,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-02 14:08:42,658 [I] connection success
2024-05-02 14:08:42,659 [I] --------- Verifying Ranger DB tables ---------
2024-05-02 14:08:42,659 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-02 14:08:42,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-02 14:08:42,892 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-02 14:08:42,893 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-02 14:08:42,893 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-02 14:08:42,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.174.84:35591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-02 14:08:43,152 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240502 14:08:43.572006 28345 mini_ranger_kms.cc:310] Created kms service
I20240502 14:08:43.782482 28345 mini_ranger_kms.cc:326] Created kudu user
I20240502 14:08:43.827224 28345 mini_ranger_kms.cc:343] Created rangerkms user
May 02 14:08:43 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:44.202569 28345 mini_ranger_kms.cc:384] Added ranger policy
I20240502 14:08:44.203097 28345 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTLmxo8/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTLmxo8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240502 14:08:44.203287 28345 mini_ranger_kms.cc:240] Using host: 127.27.174.84
I20240502 14:08:44.218631 28345 mini_ranger_kms.cc:276] Ranger KMS PID: 6782
I20240502 14:08:44.218876 28345 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.174.84:36829
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp] .....
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-kms/ews/webapp].
May 02, 2024 2:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240502 14:08:46.208282 28345 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.494s user 0.322s sys 0.550s
I20240502 14:08:46.208837 28345 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240502 14:08:46.209272 28345 mini_ranger_kms.cc:401] 127.27.174.84:36829/kms/v1/keys
May 02 14:08:46 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714658926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 02 14:08:46 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:08:47 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.174.84@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:08:49 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:08:49.200606 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33845
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33845
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:49.478770 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:49.479274 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:49.479734 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:49.507468 6834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:08:49.507800 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:49.508028 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:49.508309 6834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:08:49.508510 6834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:08:49.542940 6834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.174.84:36829/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33845
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:49.544003 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:49.544967 6834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:49.560165 6840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:49.560637 6841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:49.561316 6843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:49.562355 6834 server_base.cc:1034] running on GCE node
I20240502 14:08:50.591923 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:50.595005 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:50.596416 6834 hybrid_clock.cc:648] HybridClock initialized: now 1714658930596375 us; error 71 us; skew 500 ppm
I20240502 14:08:50.597210 6834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:50 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658930, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:50.607194 6834 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:08:50.612640 6834 webserver.cc:460] Webserver started at http://127.27.174.126:44015/ using document root <none> and password file <none>
I20240502 14:08:50.613500 6834 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:50.613684 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:50.614126 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:50 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658930, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:50.670213 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "a42ccfd1d99748998383f8032acf9ebc"
format_stamp: "Formatted at 2024-05-02 14:08:50 on dist-test-slave-dlb8"
server_key: "fc0d7e450a543f192c379d0f8f770933"
server_key_iv: "ae313aa1c035c92d8e95196b350735ea"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:50.671309 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "a42ccfd1d99748998383f8032acf9ebc"
format_stamp: "Formatted at 2024-05-02 14:08:50 on dist-test-slave-dlb8"
server_key: "fc0d7e450a543f192c379d0f8f770933"
server_key_iv: "ae313aa1c035c92d8e95196b350735ea"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:50.678493 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:08:50.706202 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:50.707405 6834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:08:50.707798 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "a42ccfd1d99748998383f8032acf9ebc"
format_stamp: "Formatted at 2024-05-02 14:08:50 on dist-test-slave-dlb8"
server_key: "fc0d7e450a543f192c379d0f8f770933"
server_key_iv: "ae313aa1c035c92d8e95196b350735ea"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:50.708283 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:50.761286 6834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:50.765861 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:50.766291 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:50.830912 6834 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33845
I20240502 14:08:50.830988 6902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33845 every 8 connection(s)
I20240502 14:08:50.833513 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:08:50.837600 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 6834
I20240502 14:08:50.838205 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:08:50.839102 6903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:08:50.839944 28345 external_mini_cluster.cc:1428] Setting key d627546f207e1533061db725a55d2319
May 02 14:08:50 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:08:50.866204 6903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Bootstrap starting.
I20240502 14:08:50.871460 6903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Neither blocks nor log segments found. Creating new log.
I20240502 14:08:50.873577 6903 log.cc:824] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:08:50.874456 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:50.842968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.27.174.126:33845)
0502 14:08:50.845737 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:50.845762 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:50.845813 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:08:50.849833 (+ 4020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:50.849872 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:50.850291 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:50.850638 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:50.854599 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:50.855918 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:50.857863 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:50.858296 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:50.864367 (+ 6071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:50.864473 (+ 106us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:50.864501 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:50.864596 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:50.868282 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:50.869657 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:50.869677 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:50.869709 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:50.869866 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:50.870471 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:50.870488 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:50.870505 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:50.871020 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:50.871345 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:50.872099 (+ 754us) server_negotiation.cc:299] Negotiation successful
0502 14:08:50.872506 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2272,"thread_start_us":1778,"threads_started":1}
I20240502 14:08:50.878700 6903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: No bootstrap required, opened a new log
I20240502 14:08:50.896775 6903 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:50.897361 6903 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:08:50.897583 6903 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42ccfd1d99748998383f8032acf9ebc, State: Initialized, Role: FOLLOWER
I20240502 14:08:50.898217 6903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:50.898650 6903 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:08:50.898909 6903 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:08:50.899168 6903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:08:50.903048 6903 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:50.903730 6903 leader_election.cc:302] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42ccfd1d99748998383f8032acf9ebc; no voters:
I20240502 14:08:50.905491 6903 leader_election.cc:288] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:08:50.905864 6908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:08:50.907964 6908 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 LEADER]: Becoming Leader. State: Replica: a42ccfd1d99748998383f8032acf9ebc, State: Running, Role: LEADER
I20240502 14:08:50.908738 6908 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:50.909631 6903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:08:50.919170 6910 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42ccfd1d99748998383f8032acf9ebc. Latest consensus state: current_term: 1 leader_uuid: "a42ccfd1d99748998383f8032acf9ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } } }
I20240502 14:08:50.919706 6910 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: This master's current role is: LEADER
I20240502 14:08:50.921236 6909 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42ccfd1d99748998383f8032acf9ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } } }
I20240502 14:08:50.921932 6909 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: This master's current role is: LEADER
I20240502 14:08:50.923516 6916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:08:50.933593 6916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:08:50.939796 6903 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240502 14:08:50.963088 6916 catalog_manager.cc:1347] Generated new cluster ID: e0c9a46b96554f0c87cb9305631ca2eb
I20240502 14:08:50.963486 6916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:08:50.997817 6916 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:08:51.000026 6916 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:08:51.018172 6916 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Generated new TSK 0
I20240502 14:08:51.019246 6916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:08:51.793184 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:08:52 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714658932, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240502 14:08:52.239034 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:52.239769 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:52.240871 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:52.296154 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:52.296721 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:52.298318 6948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:08:52.359688 6948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:52.361416 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:52.362987 6948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:52.384333 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
May 02 14:08:52 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658932, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:08:53 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658932, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
W20240502 14:08:52.385259 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:53.779268 6958 debug-util.cc:398] Leaking SignalData structure 0x7b080001e200 after lost signal to thread 6948
W20240502 14:08:53.793203 6948 thread.cc:636] GCE (cloud detector) Time spent creating pthread: real 1.415s user 0.548s sys 0.684s
W20240502 14:08:53.794459 6948 thread.cc:602] GCE (cloud detector) Time spent starting thread: real 1.416s user 0.548s sys 0.684s
W20240502 14:08:53.807910 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:53.808678 6948 server_base.cc:1034] running on GCE node
I20240502 14:08:53.810315 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:53.826146 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:53.845911 6948 hybrid_clock.cc:648] HybridClock initialized: now 1714658933845839 us; error 68 us; skew 500 ppm
I20240502 14:08:53.847159 6948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:53 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:53.858999 6948 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:08:53.886842 6948 webserver.cc:460] Webserver started at http://127.27.174.65:37271/ using document root <none> and password file <none>
I20240502 14:08:53.888617 6948 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:53.888916 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:53.889462 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:53 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:53.919124 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "759bee2ad5bd4995aa73d8e782723491"
format_stamp: "Formatted at 2024-05-02 14:08:53 on dist-test-slave-dlb8"
server_key: "fef12e8f96c18e6deef7fa77b104b281"
server_key_iv: "2e06a122fa561842d724cfb45c2719e5"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:53.920717 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "759bee2ad5bd4995aa73d8e782723491"
format_stamp: "Formatted at 2024-05-02 14:08:53 on dist-test-slave-dlb8"
server_key: "fef12e8f96c18e6deef7fa77b104b281"
server_key_iv: "2e06a122fa561842d724cfb45c2719e5"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:53.936968 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.012s sys 0.000s
I20240502 14:08:53.978963 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:53.980391 6948 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.007s
I20240502 14:08:53.980839 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "759bee2ad5bd4995aa73d8e782723491"
format_stamp: "Formatted at 2024-05-02 14:08:53 on dist-test-slave-dlb8"
server_key: "fef12e8f96c18e6deef7fa77b104b281"
server_key_iv: "2e06a122fa561842d724cfb45c2719e5"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:53.981406 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:54.069809 6948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:54.076884 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:54.077497 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:54.086222 6948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:54.094684 6948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:54.095014 6948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240502 14:08:54.095367 6948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:54.095611 6948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:54.297526 6948 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44477
I20240502 14:08:54.297580 7091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44477 every 8 connection(s)
I20240502 14:08:54.300809 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:08:54.305119 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 6948
I20240502 14:08:54.305668 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:08:54.307271 28345 external_mini_cluster.cc:1428] Setting key d4db04a5bceba447c4ddd05d9b2e98ab
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
May 02 14:08:54 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:08:54.343560 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:54.309578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.65:34519)
0502 14:08:54.314541 (+ 4963us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:54.314632 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:54.320679 (+ 6047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:54.321930 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:54.321967 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:54.323016 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:54.324353 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:54.324415 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:54.326711 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:54.326731 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:54.327672 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:54.327711 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:54.328009 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:54.328881 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:54.328947 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:54.331748 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:54.336122 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:54.336161 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:54.336216 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:54.337257 (+ 1041us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:54.337846 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:54.337882 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:54.337913 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:54.338260 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:54.339746 (+ 1486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:54.339791 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:54.340449 (+ 658us) client_negotiation.cc:769] Sending connection context
0502 14:08:54.340908 (+ 459us) client_negotiation.cc:240] Negotiation successful
0502 14:08:54.341529 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4190,"thread_start_us":3605,"threads_started":1}
I20240502 14:08:54.344278 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:54.309340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:34519 (local address 127.27.174.126:33845)
0502 14:08:54.312852 (+ 3512us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:54.312882 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:54.314819 (+ 1937us) server_negotiation.cc:407] Connection header received
0502 14:08:54.321094 (+ 6275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:54.321129 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:54.321389 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:54.321587 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:54.324711 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:54.326478 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:54.327912 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:54.328577 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:54.332043 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:54.332150 (+ 107us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:54.332169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:54.332279 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:54.335876 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:54.337476 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:54.337500 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:54.337523 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:54.337666 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:54.338521 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:54.338551 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:54.338572 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:54.339371 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:54.339761 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:54.341126 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0502 14:08:54.341743 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2914,"thread_start_us":3468,"threads_started":1}
I20240502 14:08:54.346117 7092 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:08:54.346627 7092 heartbeater.cc:464] Registering TS with master...
I20240502 14:08:54.347959 7092 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:08:54.350984 6868 ts_manager.cc:194] Registered new tserver with Master: 759bee2ad5bd4995aa73d8e782723491 (127.27.174.65:44477)
I20240502 14:08:54.352900 6868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:34519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:08:54.364385 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:54.653555 7098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:54.654109 7098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:54.654718 7098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:54.683477 7098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:54.684010 7098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:54.685873 7098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:08:54.722846 7098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:54.724066 7098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:54.725102 7098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:54.741530 7104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:55.356279 7092 heartbeater.cc:503] Master 127.27.174.126:33845 was elected leader, sending a full tablet report...
W20240502 14:08:54.741804 7105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:54.742611 7107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:54.743700 7098 server_base.cc:1034] running on GCE node
I20240502 14:08:55.798321 7098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:55.800496 7098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:55.801877 7098 hybrid_clock.cc:648] HybridClock initialized: now 1714658935801833 us; error 58 us; skew 500 ppm
I20240502 14:08:55.802685 7098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:55 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658935, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:55.814288 7098 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:08:55.819386 7098 webserver.cc:460] Webserver started at http://127.27.174.66:45379/ using document root <none> and password file <none>
I20240502 14:08:55.820304 7098 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:55.820556 7098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:55.821066 7098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:55 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658935, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:55.845121 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "be3e51d8e27d4dfd8e664aa4336fee9f"
format_stamp: "Formatted at 2024-05-02 14:08:55 on dist-test-slave-dlb8"
server_key: "3c2d843eab28646a907a9f40ea799c18"
server_key_iv: "1b193171c0089deb2dbd1586f99826fb"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:55.846400 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "be3e51d8e27d4dfd8e664aa4336fee9f"
format_stamp: "Formatted at 2024-05-02 14:08:55 on dist-test-slave-dlb8"
server_key: "3c2d843eab28646a907a9f40ea799c18"
server_key_iv: "1b193171c0089deb2dbd1586f99826fb"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:55.853771 7098 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:08:55.873396 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:55.874588 7098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:08:55.874886 7098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "be3e51d8e27d4dfd8e664aa4336fee9f"
format_stamp: "Formatted at 2024-05-02 14:08:55 on dist-test-slave-dlb8"
server_key: "3c2d843eab28646a907a9f40ea799c18"
server_key_iv: "1b193171c0089deb2dbd1586f99826fb"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:55.875233 7098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:55.927057 7098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:55.932106 7098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:55.932569 7098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:55.934959 7098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:55.938767 7098 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:55.938977 7098 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:55.939218 7098 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:55.939370 7098 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:56.069218 7098 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44601
I20240502 14:08:56.069324 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44601 every 8 connection(s)
I20240502 14:08:56.071710 7098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:08:56.081364 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7098
I20240502 14:08:56.081979 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:08:56.083422 28345 external_mini_cluster.cc:1428] Setting key 1607ae1481024e40ba50b56ac053b632
May 02 14:08:56 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658935, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:08:56.105967 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:56.075743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:48487 (local address 127.27.174.126:33845)
0502 14:08:56.076876 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:56.076891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:56.079116 (+ 2225us) server_negotiation.cc:407] Connection header received
0502 14:08:56.085417 (+ 6301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:56.085438 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:56.085616 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:56.085780 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:56.088155 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:56.089526 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:56.090913 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:56.091507 (+ 594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:56.095371 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:56.095443 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:56.095460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:56.095560 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:56.099455 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:56.101069 (+ 1614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:56.101090 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:56.101104 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:56.101232 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:56.102030 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:56.102081 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:56.102103 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:56.102495 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:56.102686 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:56.104050 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0502 14:08:56.104384 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":454,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:08:56.107745 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:56.076018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.66:48487)
0502 14:08:56.078897 (+ 2879us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:56.078964 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:56.085015 (+ 6051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:56.086009 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:56.086063 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:56.086830 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:56.087862 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:56.087907 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:56.089781 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:56.089802 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:56.090666 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:56.090703 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:56.090982 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:56.091808 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:56.091870 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:56.095062 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:56.099709 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:56.099769 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:56.099815 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:56.100847 (+ 1032us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:56.101432 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:56.101479 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:56.101509 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:56.101821 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:56.102729 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:56.102776 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:56.103408 (+ 632us) client_negotiation.cc:769] Sending connection context
0502 14:08:56.103862 (+ 454us) client_negotiation.cc:240] Negotiation successful
0502 14:08:56.104519 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2236,"thread_start_us":1611,"threads_started":1}
I20240502 14:08:56.109823 7228 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:08:56.110255 7228 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:08:56.111392 7228 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:08:56.113723 6868 ts_manager.cc:194] Registered new tserver with Master: be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601)
I20240502 14:08:56.115561 6868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:48487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:08:56.138321 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:56.440650 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:56.441154 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:56.441812 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:56.472059 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:56.472465 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:56.473379 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:08:56.506795 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:56.507951 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:56.508971 7234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:56.524652 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:57.118683 7228 heartbeater.cc:503] Master 127.27.174.126:33845 was elected leader, sending a full tablet report...
W20240502 14:08:56.525369 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:56.525755 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:56.526400 7234 server_base.cc:1034] running on GCE node
I20240502 14:08:57.573972 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:57.576227 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:57.577556 7234 hybrid_clock.cc:648] HybridClock initialized: now 1714658937577520 us; error 57 us; skew 500 ppm
I20240502 14:08:57.578356 7234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:57 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658937, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:57.584079 7234 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:08:57.588189 7234 webserver.cc:460] Webserver started at http://127.27.174.67:43719/ using document root <none> and password file <none>
I20240502 14:08:57.589010 7234 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:57.589196 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:08:57.589607 7234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 02 14:08:57 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658937, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:57.610257 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "7db773eee5cc4c5f9877670ae051432f"
format_stamp: "Formatted at 2024-05-02 14:08:57 on dist-test-slave-dlb8"
server_key: "55b62e5352a2123d88856a5ae083c87a"
server_key_iv: "b42f48c13969beb76f0d1e60f2481369"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:57.611522 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "7db773eee5cc4c5f9877670ae051432f"
format_stamp: "Formatted at 2024-05-02 14:08:57 on dist-test-slave-dlb8"
server_key: "55b62e5352a2123d88856a5ae083c87a"
server_key_iv: "b42f48c13969beb76f0d1e60f2481369"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:57.619028 7234 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20240502 14:08:57.640736 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:57.642124 7234 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20240502 14:08:57.642558 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "7db773eee5cc4c5f9877670ae051432f"
format_stamp: "Formatted at 2024-05-02 14:08:57 on dist-test-slave-dlb8"
server_key: "55b62e5352a2123d88856a5ae083c87a"
server_key_iv: "b42f48c13969beb76f0d1e60f2481369"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:57.643061 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:57.697939 7234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:57.702930 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:57.703408 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:57.705901 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:08:57.709900 7234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:08:57.710124 7234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240502 14:08:57.710378 7234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:08:57.710551 7234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:57.842264 7234 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:43605
I20240502 14:08:57.842427 7364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:43605 every 8 connection(s)
I20240502 14:08:57.844686 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:08:57.846043 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7234
I20240502 14:08:57.846777 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:08:57.848254 28345 external_mini_cluster.cc:1428] Setting key 7f9c047978883817a2af4070caa9e250
May 02 14:08:57 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658937, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:08:57.876874 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:57.849825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:38675 (local address 127.27.174.126:33845)
0502 14:08:57.853269 (+ 3444us) server_negotiation.cc:206] Beginning negotiation
0502 14:08:57.853298 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:08:57.853846 (+ 548us) server_negotiation.cc:407] Connection header received
0502 14:08:57.858547 (+ 4701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:08:57.858572 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:08:57.858737 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:08:57.858870 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:08:57.860931 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:57.861761 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:08:57.863163 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:08:57.863730 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:57.867011 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:08:57.867107 (+ 96us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:08:57.867135 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:08:57.867265 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:08:57.871156 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:57.872114 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:57.872139 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:57.872152 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:57.872321 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:08:57.872915 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:08:57.872934 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:08:57.872948 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:08:57.873325 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:08:57.873580 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0502 14:08:57.874464 (+ 884us) server_negotiation.cc:299] Negotiation successful
0502 14:08:57.874876 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2832,"thread_start_us":2541,"threads_started":1}
I20240502 14:08:57.877437 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:08:57.850070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.67:38675)
0502 14:08:57.853594 (+ 3524us) negotiation.cc:107] Waiting for socket to connect
0502 14:08:57.853683 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0502 14:08:57.858225 (+ 4542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:08:57.859158 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:08:57.859182 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:08:57.859809 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:08:57.860697 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:57.860733 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:57.862064 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:08:57.862085 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:08:57.862913 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:08:57.862950 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:08:57.863323 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:08:57.864171 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:08:57.864232 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:08:57.866735 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:08:57.871359 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:57.871383 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:57.871404 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:57.871937 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:57.872500 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:08:57.872520 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:08:57.872534 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:08:57.872730 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:08:57.873551 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:08:57.873583 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:08:57.873939 (+ 356us) client_negotiation.cc:769] Sending connection context
0502 14:08:57.874282 (+ 343us) client_negotiation.cc:240] Negotiation successful
0502 14:08:57.874786 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2795,"thread_start_us":2258,"threads_started":1}
I20240502 14:08:57.878932 7365 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:08:57.879254 7365 heartbeater.cc:464] Registering TS with master...
I20240502 14:08:57.880165 7365 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:08:57.882433 6868 ts_manager.cc:194] Registered new tserver with Master: 7db773eee5cc4c5f9877670ae051432f (127.27.174.67:43605)
I20240502 14:08:57.883957 6868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:38675
I20240502 14:08:57.884666 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:08:57.893371 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 6948
I20240502 14:08:57.914299 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7098
I20240502 14:08:57.935894 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7234
I20240502 14:08:57.957186 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 6834
I20240502 14:08:57.993347 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33845
--webserver_interface=127.27.174.126
--webserver_port=44015
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33845
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:08:58.277870 7369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:08:58.278491 7369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:08:58.279064 7369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:08:58.311443 7369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:08:58.311968 7369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:08:58.312426 7369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:08:58.312811 7369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:08:58.313165 7369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:08:58.372200 7369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33845
--ranger_config_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=44015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:08:58.373417 7369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:08:58.374504 7369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:08:58.390075 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:08:58.391315 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:58.394534 7369 server_base.cc:1034] running on GCE node
W20240502 14:08:58.394847 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:08:59.443835 7369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:08:59.446416 7369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:08:59.447827 7369 hybrid_clock.cc:648] HybridClock initialized: now 1714658939447778 us; error 56 us; skew 500 ppm
I20240502 14:08:59.448614 7369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:08:59 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:08:59.454440 7369 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:08:59.459277 7369 webserver.cc:460] Webserver started at http://127.27.174.126:44015/ using document root <none> and password file <none>
I20240502 14:08:59.460130 7369 fs_manager.cc:359] Metadata directory not provided
I20240502 14:08:59.460340 7369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 02 14:08:59 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:08:59.515931 7369 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 14:08:59.520510 7387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:08:59.521701 7369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:08:59.522130 7369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "a42ccfd1d99748998383f8032acf9ebc"
format_stamp: "Formatted at 2024-05-02 14:08:50 on dist-test-slave-dlb8"
server_key: "fc0d7e450a543f192c379d0f8f770933"
server_key_iv: "ae313aa1c035c92d8e95196b350735ea"
server_key_version: "kuduclusterkey@0"
I20240502 14:08:59.524420 7369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:08:59.576390 7369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:08:59.581171 7369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:08:59.581591 7369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:08:59.646723 7369 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33845
I20240502 14:08:59.646854 7438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33845 every 8 connection(s)
I20240502 14:08:59.649536 7369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:08:59.656262 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7369
I20240502 14:08:59.658788 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:44477
--local_ip_for_outbound_sockets=127.27.174.65
--tserver_master_addrs=127.27.174.126:33845
--webserver_port=37271
--webserver_interface=127.27.174.65
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240502 14:08:59.661442 7439 sys_catalog.cc:263] Verifying existing consensus state
I20240502 14:08:59.668831 7439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Bootstrap starting.
I20240502 14:08:59.708307 7439 log.cc:824] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:08:59.724416 7439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:08:59.725188 7439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Bootstrap complete.
I20240502 14:08:59.748445 7439 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:59.750694 7439 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42ccfd1d99748998383f8032acf9ebc, State: Initialized, Role: FOLLOWER
I20240502 14:08:59.751524 7439 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:59.751998 7439 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:08:59.752310 7439 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:08:59.752636 7439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:08:59.759078 7439 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:59.759805 7439 leader_election.cc:302] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42ccfd1d99748998383f8032acf9ebc; no voters:
I20240502 14:08:59.762118 7439 leader_election.cc:288] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 14:08:59.762544 7443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:08:59.765738 7443 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [term 2 LEADER]: Becoming Leader. State: Replica: a42ccfd1d99748998383f8032acf9ebc, State: Running, Role: LEADER
I20240502 14:08:59.766646 7443 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } }
I20240502 14:08:59.767369 7439 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:08:59.777272 7444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a42ccfd1d99748998383f8032acf9ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } } }
I20240502 14:08:59.777961 7444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: This master's current role is: LEADER
I20240502 14:08:59.780167 7445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42ccfd1d99748998383f8032acf9ebc. Latest consensus state: current_term: 2 leader_uuid: "a42ccfd1d99748998383f8032acf9ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42ccfd1d99748998383f8032acf9ebc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33845 } } }
I20240502 14:08:59.780814 7445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc [sys.catalog]: This master's current role is: LEADER
I20240502 14:08:59.788071 7450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:08:59.801091 7450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:08:59.805466 7450 catalog_manager.cc:1259] Loaded cluster ID: e0c9a46b96554f0c87cb9305631ca2eb
I20240502 14:08:59.805795 7450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:08:59.812924 7450 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:08:59.818343 7450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Loaded TSK: 0
I20240502 14:08:59.820542 7450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:08:59.829735 7439 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20240502 14:09:00.103977 7441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:00.104705 7441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:00.105770 7441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:00.159093 7441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:00.159649 7441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:00.161094 7441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:00.222548 7441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:44477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=37271
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:00.224138 7441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:00.225493 7441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:00.265875 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:00.871156 7490 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
May 02 14:09:01 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240502 14:09:00.266713 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:00.267804 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:00.267674 7441 server_base.cc:1034] running on GCE node
I20240502 14:09:01.624538 7441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:01.642917 7441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:01.644513 7441 hybrid_clock.cc:648] HybridClock initialized: now 1714658941644441 us; error 91 us; skew 500 ppm
I20240502 14:09:01.645709 7441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:01 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:01.664011 7441 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:09:01.688410 7441 webserver.cc:460] Webserver started at http://127.27.174.65:37271/ using document root <none> and password file <none>
I20240502 14:09:01.689846 7441 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:01.690249 7441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 02 14:09:01 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:09:01.732242 7441 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.007s
I20240502 14:09:01.738701 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:01.740236 7441 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20240502 14:09:01.740710 7441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "759bee2ad5bd4995aa73d8e782723491"
format_stamp: "Formatted at 2024-05-02 14:08:53 on dist-test-slave-dlb8"
server_key: "fef12e8f96c18e6deef7fa77b104b281"
server_key_iv: "2e06a122fa561842d724cfb45c2719e5"
server_key_version: "kuduclusterkey@0"
I20240502 14:09:01.744330 7441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:01.857008 7441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:01.864693 7441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:01.865356 7441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:01.874569 7441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:01.886327 7441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:01.886700 7441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:01.887089 7441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:01.887378 7441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 02 14:09:01 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
May 02 14:09:02 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:09:02.530612 7441 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:44477
I20240502 14:09:02.530759 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:44477 every 8 connection(s)
I20240502 14:09:02.534591 7441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:02.540225 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7441
I20240502 14:09:02.543000 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:44601
--local_ip_for_outbound_sockets=127.27.174.66
--tserver_master_addrs=127.27.174.126:33845
--webserver_port=45379
--webserver_interface=127.27.174.66
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:09:02 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:02.606521 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:02.544159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.65:38951)
0502 14:09:02.554135 (+ 9976us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:02.554260 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:02.561961 (+ 7701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:02.564796 (+ 2835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:02.564831 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:02.565866 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:02.567228 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:02.567294 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:02.570090 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:02.570119 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:02.571437 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:02.571491 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:02.573065 (+ 1574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:02.574213 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:02.574303 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:02.584978 (+ 10675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:02.595065 (+ 10087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:02.595096 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:02.595124 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:02.596085 (+ 961us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:02.599644 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:02.599671 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:02.599693 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:02.599999 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:02.601353 (+ 1354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:02.601385 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:02.601915 (+ 530us) client_negotiation.cc:769] Sending connection context
0502 14:09:02.602366 (+ 451us) client_negotiation.cc:240] Negotiation successful
0502 14:09:02.603001 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9192,"thread_start_us":8802,"threads_started":1}
I20240502 14:09:02.611398 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:02.546336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:38951 (local address 127.27.174.126:33845)
0502 14:09:02.560020 (+ 13684us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:02.560067 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:02.560149 (+ 82us) server_negotiation.cc:407] Connection header received
0502 14:09:02.562957 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:02.563003 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:02.563477 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:02.564024 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:02.567710 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:02.569739 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:02.571864 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:02.572480 (+ 616us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:02.585396 (+ 12916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:02.585524 (+ 128us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:02.585575 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:02.585725 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:02.594799 (+ 9074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:02.599056 (+ 4257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:02.599089 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:02.599124 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:02.599401 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:02.600285 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:02.600310 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:02.600332 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:02.601079 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:02.608041 (+ 6962us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:02.608450 (+ 409us) server_negotiation.cc:299] Negotiation successful
0502 14:09:02.609052 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12916,"thread_start_us":13681,"threads_started":1}
I20240502 14:09:02.616183 7616 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:09:02.616672 7616 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:02.617916 7616 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:09:02.620975 7404 ts_manager.cc:194] Registered new tserver with Master: 759bee2ad5bd4995aa73d8e782723491 (127.27.174.65:44477)
I20240502 14:09:02.623881 7404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:38951
W20240502 14:09:02.916977 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:02.917529 7620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:02.918133 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:02.946527 7620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:02.946873 7620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:02.947665 7620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:02.982656 7620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:44601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=45379
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:02.983915 7620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:02.984915 7620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:03.002645 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:03.631383 7616 heartbeater.cc:503] Master 127.27.174.126:33845 was elected leader, sending a full tablet report...
W20240502 14:09:03.003392 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:03.005059 7620 server_base.cc:1034] running on GCE node
W20240502 14:09:03.006417 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:04.057641 7620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:04.060122 7620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:04.061460 7620 hybrid_clock.cc:648] HybridClock initialized: now 1714658944061405 us; error 73 us; skew 500 ppm
I20240502 14:09:04.062281 7620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:04 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:04.068287 7620 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:09:04.073311 7620 webserver.cc:460] Webserver started at http://127.27.174.66:45379/ using document root <none> and password file <none>
I20240502 14:09:04.074219 7620 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:04.074430 7620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 02 14:09:04 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:09:04.097646 7620 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:09:04.101933 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:04.103009 7620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:09:04.103296 7620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "be3e51d8e27d4dfd8e664aa4336fee9f"
format_stamp: "Formatted at 2024-05-02 14:08:55 on dist-test-slave-dlb8"
server_key: "3c2d843eab28646a907a9f40ea799c18"
server_key_iv: "1b193171c0089deb2dbd1586f99826fb"
server_key_version: "kuduclusterkey@0"
I20240502 14:09:04.105438 7620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:04.172770 7620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:04.177625 7620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:04.178012 7620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:04.180316 7620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:04.184062 7620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:04.184247 7620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:04.184423 7620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:04.184548 7620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:04.308382 7620 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:44601
I20240502 14:09:04.308523 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:44601 every 8 connection(s)
I20240502 14:09:04.310963 7620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:04.313270 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7620
I20240502 14:09:04.315568 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.174.84:36829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:43605
--local_ip_for_outbound_sockets=127.27.174.67
--tserver_master_addrs=127.27.174.126:33845
--webserver_port=43719
--webserver_interface=127.27.174.67
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:09:04 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:04.344229 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:04.315509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:35237 (local address 127.27.174.126:33845)
0502 14:09:04.317138 (+ 1629us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:04.317158 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:04.318617 (+ 1459us) server_negotiation.cc:407] Connection header received
0502 14:09:04.323622 (+ 5005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:04.323642 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:04.323844 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:04.324061 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:04.326226 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:04.327685 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:04.329102 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:04.329692 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:04.333534 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:04.333614 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:04.333632 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:04.333737 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:04.337488 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:04.338946 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:04.338967 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:04.338988 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:04.339180 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:04.340011 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:04.340044 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:04.340068 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:04.340635 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:04.340892 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:04.342510 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0502 14:09:04.342884 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":712,"threads_started":1}
I20240502 14:09:04.345615 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:04.315561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.66:35237)
0502 14:09:04.318377 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:04.318438 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:04.323198 (+ 4760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:04.324288 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:04.324311 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:04.324962 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:04.325894 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:04.325947 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:04.327936 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:04.327974 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:04.328860 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:04.328897 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:04.329165 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:04.329988 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:04.330083 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:04.333202 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:04.337703 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:04.337753 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:04.337788 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:04.338737 (+ 949us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:04.339396 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:04.339421 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:04.339446 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:04.339761 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:04.341066 (+ 1305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:04.341131 (+ 65us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:04.341673 (+ 542us) client_negotiation.cc:769] Sending connection context
0502 14:09:04.342221 (+ 548us) client_negotiation.cc:240] Negotiation successful
0502 14:09:04.342890 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"thread_start_us":1649,"threads_started":1}
I20240502 14:09:04.347636 7758 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:09:04.348007 7758 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:04.348978 7758 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:09:04.351001 7404 ts_manager.cc:194] Registered new tserver with Master: be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601)
I20240502 14:09:04.352185 7404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:35237
W20240502 14:09:04.611315 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:04.611831 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:04.612452 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:04.643759 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:04.644145 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:04.645068 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:04.680661 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.174.84:36829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:43605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=43719
--tserver_master_addrs=127.27.174.126:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:04.681844 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:04.682862 7762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:04.705606 7768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:05.355063 7758 heartbeater.cc:503] Master 127.27.174.126:33845 was elected leader, sending a full tablet report...
I20240502 14:09:04.706931 7762 server_base.cc:1034] running on GCE node
W20240502 14:09:04.707697 7771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:04.707579 7769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:05.741634 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:05.743940 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:05.745298 7762 hybrid_clock.cc:648] HybridClock initialized: now 1714658945745260 us; error 70 us; skew 500 ppm
I20240502 14:09:05.746105 7762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:05 dist-test-slave-dlb8 krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658945, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:05.751871 7762 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:09:05.756997 7762 webserver.cc:460] Webserver started at http://127.27.174.67:43719/ using document root <none> and password file <none>
I20240502 14:09:05.757899 7762 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:05.758174 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 02 14:09:05 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658945, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for HTTP/127.27.174.84@KRBTEST.COM
I20240502 14:09:05.794700 7762 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 14:09:05.799501 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:05.800418 7762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20240502 14:09:05.800731 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "7db773eee5cc4c5f9877670ae051432f"
format_stamp: "Formatted at 2024-05-02 14:08:57 on dist-test-slave-dlb8"
server_key: "55b62e5352a2123d88856a5ae083c87a"
server_key_iv: "b42f48c13969beb76f0d1e60f2481369"
server_key_version: "kuduclusterkey@0"
I20240502 14:09:05.802944 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:05.847898 7762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:05.852725 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:05.853178 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:05.855650 7762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:05.859325 7762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:05.859514 7762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:05.859794 7762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:05.859930 7762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:05.985008 7762 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:43605
I20240502 14:09:05.985148 7892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:43605 every 8 connection(s)
I20240502 14:09:05.987443 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:05.994199 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7762
May 02 14:09:06 dist-test-slave-dlb8 krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658945, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:06.019232 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:05.996052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.27.174.126:33845)
0502 14:09:05.998368 (+ 2316us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:05.998396 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:05.999097 (+ 701us) server_negotiation.cc:407] Connection header received
0502 14:09:05.999535 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:05.999554 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:05.999717 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:05.999870 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:06.002738 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.003688 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.005177 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.005740 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.008695 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:06.008793 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:06.008820 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:06.008966 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:06.013783 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.015106 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.015124 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.015140 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.015266 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.015872 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.015891 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.015903 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.016254 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:06.016591 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.017281 (+ 690us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.017660 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":1450,"threads_started":1}
I20240502 14:09:06.021755 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:05.991191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34463 (local address 127.27.174.126:33845)
0502 14:09:05.992259 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:05.992277 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:05.994921 (+ 2644us) server_negotiation.cc:407] Connection header received
0502 14:09:06.000654 (+ 5733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.000674 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.000851 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.000992 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:06.003326 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.004449 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.005814 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.006353 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.009919 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:06.010002 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:06.010018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:06.010149 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:06.015123 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.016380 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.016416 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.016438 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.016626 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.017327 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.017371 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.017397 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.017912 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:06.018161 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.019171 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.019621 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20240502 14:09:06.023088 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:05.991510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33845 (local address 127.27.174.67:34463)
0502 14:09:05.994687 (+ 3177us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:05.994753 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.000313 (+ 5560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.001319 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.001355 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.002103 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:06.003037 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.003081 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.004684 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.004703 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.005577 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.005616 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.006023 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.006814 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:06.006881 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:06.009668 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:06.015337 (+ 5669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:06.015365 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:06.015400 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:06.016162 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:06.016824 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:06.016842 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:06.016856 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:06.017086 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:06.018260 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:06.018282 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:06.018628 (+ 346us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.018936 (+ 308us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.019491 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"thread_start_us":1788,"threads_started":1}
I20240502 14:09:06.024741 7893 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33845
I20240502 14:09:06.025084 7893 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:06.026013 7893 heartbeater.cc:511] Master 127.27.174.126:33845 requested a full tablet report, sending...
I20240502 14:09:06.027858 7404 ts_manager.cc:194] Registered new tserver with Master: 7db773eee5cc4c5f9877670ae051432f (127.27.174.67:43605)
I20240502 14:09:06.029008 7404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:34463
I20240502 14:09:06.036907 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:06.059979 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.042802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.27.174.126:33845)
0502 14:09:06.043323 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.043340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.044590 (+ 1250us) server_negotiation.cc:407] Connection header received
0502 14:09:06.044821 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.044835 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.044982 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.045095 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:06.047292 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.048139 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.049400 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.049713 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.052050 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:06.052117 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:06.052134 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:06.052230 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:06.055482 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.056546 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.056565 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.056582 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.056715 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:06.057161 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:06.057177 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:06.057189 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:06.057533 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:06.057793 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.058304 (+ 511us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.058620 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20240502 14:09:06.066746 7404 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:06.087218 7404 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:06.130946 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.115584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.27.174.67:43605)
0502 14:09:06.118536 (+ 2952us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.118576 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.119333 (+ 757us) server_negotiation.cc:407] Connection header received
0502 14:09:06.119798 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.119843 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.120284 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.120551 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.123655 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.125206 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.127390 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.128625 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.128861 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.129114 (+ 253us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.129358 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2103,"thread_start_us":1723,"threads_started":1}
I20240502 14:09:06.135597 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.115149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:43605 (local address 127.0.0.1:54876)
0502 14:09:06.119109 (+ 3960us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.119184 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.119495 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.121606 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.121665 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.121868 (+ 203us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.123299 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.123358 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.125450 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.125469 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.127079 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.127127 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.128659 (+ 1532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.128767 (+ 108us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.133499 (+ 4732us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.133761 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3171,"thread_start_us":3737,"threads_started":1}
I20240502 14:09:06.140200 7828 tablet_service.cc:1467] Processing CreateTablet for tablet c0459f6590484248829354aa6d479318 (DEFAULT_TABLE table=test-table [id=0b7167ffa0b6479c84052e7006bed74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:06.141564 7828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0459f6590484248829354aa6d479318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:06.141748 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.119934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44601 (local address 127.0.0.1:38734)
0502 14:09:06.131248 (+ 11314us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.131292 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.131499 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.134191 (+ 2692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.134233 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.134349 (+ 116us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.135309 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.135366 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.137527 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.137559 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.139396 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.139460 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.139750 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.139824 (+ 74us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.139932 (+ 108us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.140116 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10718,"thread_start_us":10469,"threads_started":1}
I20240502 14:09:06.142504 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.117801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44477 (local address 127.0.0.1:52426)
0502 14:09:06.129628 (+ 11827us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.129688 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.129937 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.134510 (+ 4573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.134531 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.134644 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.135506 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.135572 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.138411 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.138444 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.140099 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.140144 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.140412 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.140492 (+ 80us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.140594 (+ 102us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.140785 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11238,"thread_start_us":3408,"threads_started":1}
I20240502 14:09:06.145881 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.120243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.27.174.66:44601)
0502 14:09:06.132785 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.132815 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.132868 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:09:06.133067 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.133101 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.133496 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.133707 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.135613 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.137275 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.142387 (+ 5112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.143804 (+ 1417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.144036 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.144282 (+ 246us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.144527 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11892,"thread_start_us":11576,"threads_started":1}
I20240502 14:09:06.146524 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.117915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.27.174.65:44477)
0502 14:09:06.133567 (+ 15652us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.133599 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.133655 (+ 56us) server_negotiation.cc:407] Connection header received
0502 14:09:06.133851 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.133886 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.134199 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.135679 (+ 1480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.136114 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.138138 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.143155 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.144566 (+ 1411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.144771 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.144950 (+ 179us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.145160 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15044,"thread_start_us":14479,"threads_started":1}
I20240502 14:09:06.150530 7693 tablet_service.cc:1467] Processing CreateTablet for tablet c0459f6590484248829354aa6d479318 (DEFAULT_TABLE table=test-table [id=0b7167ffa0b6479c84052e7006bed74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:06.150523 7550 tablet_service.cc:1467] Processing CreateTablet for tablet c0459f6590484248829354aa6d479318 (DEFAULT_TABLE table=test-table [id=0b7167ffa0b6479c84052e7006bed74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:06.152330 7550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0459f6590484248829354aa6d479318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:06.152331 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0459f6590484248829354aa6d479318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:06.166049 7914 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Bootstrap starting.
I20240502 14:09:06.171034 7914 tablet_bootstrap.cc:654] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:06.172633 7914 log.cc:824] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:06.178987 7914 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: No bootstrap required, opened a new log
I20240502 14:09:06.179420 7914 ts_tablet_manager.cc:1367] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240502 14:09:06.181210 7916 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Bootstrap starting.
I20240502 14:09:06.185904 7917 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Bootstrap starting.
I20240502 14:09:06.188705 7916 tablet_bootstrap.cc:654] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:06.190681 7917 tablet_bootstrap.cc:654] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:06.192607 7917 log.cc:824] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:06.192606 7916 log.cc:824] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:06.198433 7916 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: No bootstrap required, opened a new log
I20240502 14:09:06.198721 7917 tablet_bootstrap.cc:492] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: No bootstrap required, opened a new log
I20240502 14:09:06.198817 7916 ts_tablet_manager.cc:1367] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20240502 14:09:06.199164 7917 ts_tablet_manager.cc:1367] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20240502 14:09:06.208745 7914 raft_consensus.cc:348] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.209677 7914 raft_consensus.cc:374] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:06.210011 7914 raft_consensus.cc:729] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db773eee5cc4c5f9877670ae051432f, State: Initialized, Role: FOLLOWER
I20240502 14:09:06.210824 7914 consensus_queue.cc:260] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.213804 7893 heartbeater.cc:503] Master 127.27.174.126:33845 was elected leader, sending a full tablet report...
I20240502 14:09:06.214665 7914 ts_tablet_manager.cc:1398] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Time spent starting tablet: real 0.035s user 0.026s sys 0.008s
I20240502 14:09:06.223119 7916 raft_consensus.cc:348] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.224361 7916 raft_consensus.cc:374] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:06.224697 7916 raft_consensus.cc:729] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3e51d8e27d4dfd8e664aa4336fee9f, State: Initialized, Role: FOLLOWER
I20240502 14:09:06.225659 7916 consensus_queue.cc:260] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.229046 7920 raft_consensus.cc:482] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:06.229864 7916 ts_tablet_manager.cc:1398] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Time spent starting tablet: real 0.031s user 0.025s sys 0.004s
I20240502 14:09:06.229590 7920 raft_consensus.cc:504] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.230963 7917 raft_consensus.cc:348] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.231792 7917 raft_consensus.cc:374] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:06.232065 7917 raft_consensus.cc:729] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759bee2ad5bd4995aa73d8e782723491, State: Initialized, Role: FOLLOWER
I20240502 14:09:06.232932 7917 consensus_queue.cc:260] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.236546 7920 leader_election.cc:288] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601), 759bee2ad5bd4995aa73d8e782723491 (127.27.174.65:44477)
W20240502 14:09:06.241597 7894 tablet.cc:2369] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:06.245793 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.232590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:40409 (local address 127.27.174.66:44601)
0502 14:09:06.233360 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.233390 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.233448 (+ 58us) server_negotiation.cc:407] Connection header received
0502 14:09:06.233806 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.233833 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.234119 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.234325 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.235949 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.237568 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.241890 (+ 4322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.243491 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.243689 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.243929 (+ 240us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.244213 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20240502 14:09:06.246923 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.232388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44601 (local address 127.27.174.67:40409)
0502 14:09:06.233259 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.233302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.233534 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.234745 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.234794 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.234926 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.235512 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.235572 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.237881 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.237908 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.239774 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.239836 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.240230 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.240347 (+ 117us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.240506 (+ 159us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.240740 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20240502 14:09:06.247498 7917 ts_tablet_manager.cc:1398] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Time spent starting tablet: real 0.048s user 0.040s sys 0.005s
I20240502 14:09:06.249125 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "7db773eee5cc4c5f9877670ae051432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" is_pre_election: true
I20240502 14:09:06.249948 7713 raft_consensus.cc:2454] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db773eee5cc4c5f9877670ae051432f in term 0.
I20240502 14:09:06.251084 7782 leader_election.cc:302] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db773eee5cc4c5f9877670ae051432f, be3e51d8e27d4dfd8e664aa4336fee9f; no voters:
I20240502 14:09:06.251931 7920 raft_consensus.cc:2781] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:06.252302 7920 raft_consensus.cc:482] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:06.252977 7920 raft_consensus.cc:3037] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:06.254348 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.234750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44477 (local address 127.27.174.67:33787)
0502 14:09:06.245739 (+ 10989us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.245787 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.246151 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.247200 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.247226 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.247336 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.248375 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.248429 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.250000 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.250030 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.251713 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.251751 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.252110 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.252211 (+ 101us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.252346 (+ 135us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.252528 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10343,"thread_start_us":10022,"threads_started":1}
I20240502 14:09:06.255415 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.237545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33787 (local address 127.27.174.65:44477)
0502 14:09:06.238237 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.238264 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.245908 (+ 7644us) server_negotiation.cc:407] Connection header received
0502 14:09:06.246451 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.246478 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.246702 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.246828 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.248649 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.249761 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.252029 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.252965 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.253113 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.253288 (+ 175us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.253484 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:09:06.255959 7922 raft_consensus.cc:482] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:06.257404 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "7db773eee5cc4c5f9877670ae051432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759bee2ad5bd4995aa73d8e782723491" is_pre_election: true
I20240502 14:09:06.256630 7922 raft_consensus.cc:504] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.258697 7570 raft_consensus.cc:2454] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db773eee5cc4c5f9877670ae051432f in term 0.
I20240502 14:09:06.260457 7922 leader_election.cc:288] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db773eee5cc4c5f9877670ae051432f (127.27.174.67:43605), be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601)
I20240502 14:09:06.260768 7920 raft_consensus.cc:504] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.263523 7920 leader_election.cc:288] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [CANDIDATE]: Term 1 election: Requested vote from peers be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601), 759bee2ad5bd4995aa73d8e782723491 (127.27.174.65:44477)
I20240502 14:09:06.264484 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "7db773eee5cc4c5f9877670ae051432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f"
I20240502 14:09:06.265019 7713 raft_consensus.cc:3037] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:06.265393 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "7db773eee5cc4c5f9877670ae051432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759bee2ad5bd4995aa73d8e782723491"
I20240502 14:09:06.266074 7570 raft_consensus.cc:3037] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:06.273424 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.260789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33189 (local address 127.27.174.67:43605)
0502 14:09:06.261485 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.261518 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.263430 (+ 1912us) server_negotiation.cc:407] Connection header received
0502 14:09:06.263953 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.263984 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.264234 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.264366 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.266256 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.267452 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.270007 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.271157 (+ 1150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.271312 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.271932 (+ 620us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.272162 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20240502 14:09:06.273892 7713 raft_consensus.cc:2454] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db773eee5cc4c5f9877670ae051432f in term 1.
I20240502 14:09:06.275029 7782 leader_election.cc:302] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db773eee5cc4c5f9877670ae051432f, be3e51d8e27d4dfd8e664aa4336fee9f; no voters:
I20240502 14:09:06.275885 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.260619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:43605 (local address 127.27.174.65:33189)
0502 14:09:06.263255 (+ 2636us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.263311 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.263678 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.264789 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.264817 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.264908 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.265927 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.265988 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.267813 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.267843 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.269699 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.269752 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.271557 (+ 1805us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.271642 (+ 85us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.273930 (+ 2288us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.274157 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968,"thread_start_us":1199,"threads_started":1}
I20240502 14:09:06.275884 7920 raft_consensus.cc:2781] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:06.277107 7570 raft_consensus.cc:2454] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db773eee5cc4c5f9877670ae051432f in term 1.
I20240502 14:09:06.280534 7920 raft_consensus.cc:686] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 1 LEADER]: Becoming Leader. State: Replica: 7db773eee5cc4c5f9877670ae051432f, State: Running, Role: LEADER
I20240502 14:09:06.280957 7848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "759bee2ad5bd4995aa73d8e782723491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db773eee5cc4c5f9877670ae051432f" is_pre_election: true
I20240502 14:09:06.281553 7920 consensus_queue.cc:237] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } }
I20240502 14:09:06.285701 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.262641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:44601 (local address 127.27.174.65:59673)
0502 14:09:06.277017 (+ 14376us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.277067 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:06.277455 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:06.278828 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:06.278871 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:06.278999 (+ 128us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:06.279632 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.279702 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.281608 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.281642 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:06.283242 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:06.283295 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.283602 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.283692 (+ 90us) client_negotiation.cc:769] Sending connection context
0502 14:09:06.283830 (+ 138us) client_negotiation.cc:240] Negotiation successful
0502 14:09:06.284008 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13718,"thread_start_us":16610,"threads_started":1}
I20240502 14:09:06.286281 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.262924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:59673 (local address 127.27.174.66:44601)
0502 14:09:06.271663 (+ 8739us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.271707 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.277199 (+ 5492us) server_negotiation.cc:407] Connection header received
0502 14:09:06.277796 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.277838 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.278198 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.278366 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:06.280012 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.281294 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.283497 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.284317 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.284425 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.284623 (+ 198us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.284813 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20240502 14:09:06.290426 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0459f6590484248829354aa6d479318" candidate_uuid: "759bee2ad5bd4995aa73d8e782723491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" is_pre_election: true
I20240502 14:09:06.291235 7713 raft_consensus.cc:2379] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 759bee2ad5bd4995aa73d8e782723491 in current term 1: Already voted for candidate 7db773eee5cc4c5f9877670ae051432f in this term.
I20240502 14:09:06.292249 7504 leader_election.cc:302] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 759bee2ad5bd4995aa73d8e782723491; no voters: 7db773eee5cc4c5f9877670ae051432f, be3e51d8e27d4dfd8e664aa4336fee9f
I20240502 14:09:06.293012 7922 raft_consensus.cc:2726] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:09:06.295364 7404 catalog_manager.cc:5484] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7db773eee5cc4c5f9877670ae051432f (127.27.174.67). New cstate: current_term: 1 leader_uuid: "7db773eee5cc4c5f9877670ae051432f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db773eee5cc4c5f9877670ae051432f" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 } health_report { overall_health: UNKNOWN } } }
W20240502 14:09:06.316212 7617 tablet.cc:2369] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:09:06.316955 7759 tablet.cc:2369] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:06.346041 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.338148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.27.174.67:43605)
0502 14:09:06.338763 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:06.338782 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:06.338913 (+ 131us) server_negotiation.cc:407] Connection header received
0502 14:09:06.339341 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:06.339359 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:06.339628 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:06.339784 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:06.340889 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.341616 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:06.343332 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:06.343678 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:06.343853 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:06.344481 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:06.344624 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:06.344836 (+ 212us) server_negotiation.cc:299] Negotiation successful
0502 14:09:06.344951 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20240502 14:09:06.357687 7570 raft_consensus.cc:1261] T c0459f6590484248829354aa6d479318 P 759bee2ad5bd4995aa73d8e782723491 [term 1 FOLLOWER]: Refusing update from remote peer 7db773eee5cc4c5f9877670ae051432f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:06.357906 7713 raft_consensus.cc:1261] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 1 FOLLOWER]: Refusing update from remote peer 7db773eee5cc4c5f9877670ae051432f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:06.359208 7926 consensus_queue.cc:1035] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [LEADER]: Connected to new peer: Peer: permanent_uuid: "759bee2ad5bd4995aa73d8e782723491" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:06.359766 7920 consensus_queue.cc:1035] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [LEADER]: Connected to new peer: Peer: permanent_uuid: "be3e51d8e27d4dfd8e664aa4336fee9f" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:06.380861 7929 mvcc.cc:205] Tried to move back new op lower bound from 7023243044266504192 to 7023243043979313152. Current Snapshot: MvccSnapshot[applied={T|T < 7023243044266504192}]
I20240502 14:09:06.414741 7403 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:06.415410 7403 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:06.425940 7403 catalog_manager.cc:5771] T 00000000000000000000000000000000 P a42ccfd1d99748998383f8032acf9ebc: Sending DeleteTablet for 3 replicas of tablet c0459f6590484248829354aa6d479318
I20240502 14:09:06.427923 7828 tablet_service.cc:1514] Processing DeleteTablet for tablet c0459f6590484248829354aa6d479318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:54876
I20240502 14:09:06.428220 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet c0459f6590484248829354aa6d479318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:38734
I20240502 14:09:06.428999 7550 tablet_service.cc:1514] Processing DeleteTablet for tablet c0459f6590484248829354aa6d479318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:06 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52426
I20240502 14:09:06.435345 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7441
I20240502 14:09:06.452117 7940 tablet_replica.cc:330] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: stopping tablet replica
I20240502 14:09:06.453141 7940 raft_consensus.cc:2229] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:06.453963 7940 raft_consensus.cc:2258] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:06.455821 7939 tablet_replica.cc:330] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: stopping tablet replica
I20240502 14:09:06.456975 7939 raft_consensus.cc:2229] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:06.457604 7940 ts_tablet_manager.cc:1875] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:06.458002 7939 raft_consensus.cc:2258] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:06.461323 7939 ts_tablet_manager.cc:1875] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:06.472275 7940 ts_tablet_manager.cc:1888] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:06.472811 7940 log.cc:1196] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/c0459f6590484248829354aa6d479318
I20240502 14:09:06.473775 7940 ts_tablet_manager.cc:1909] T c0459f6590484248829354aa6d479318 P be3e51d8e27d4dfd8e664aa4336fee9f: Deleting consensus metadata
W20240502 14:09:06.476158 7390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:44477 (error 108)
I20240502 14:09:06.476315 7939 ts_tablet_manager.cc:1888] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:06.476687 7939 log.cc:1196] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/c0459f6590484248829354aa6d479318
I20240502 14:09:06.477355 7389 catalog_manager.cc:4830] TS be3e51d8e27d4dfd8e664aa4336fee9f (127.27.174.66:44601): tablet c0459f6590484248829354aa6d479318 (table test-table [id=0b7167ffa0b6479c84052e7006bed74e]) successfully deleted
I20240502 14:09:06.477803 7939 ts_tablet_manager.cc:1909] T c0459f6590484248829354aa6d479318 P 7db773eee5cc4c5f9877670ae051432f: Deleting consensus metadata
I20240502 14:09:06.481289 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7620
I20240502 14:09:06.481446 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:06.479971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:44477 (local address 127.0.0.1:52442)
0502 14:09:06.480635 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:06.480881 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20240502 14:09:06.483088 7389 catalog_manager.cc:4830] TS 7db773eee5cc4c5f9877670ae051432f (127.27.174.67:43605): tablet c0459f6590484248829354aa6d479318 (table test-table [id=0b7167ffa0b6479c84052e7006bed74e]) successfully deleted
W20240502 14:09:06.482941 7390 catalog_manager.cc:4573] TS 759bee2ad5bd4995aa73d8e782723491 (127.27.174.65:44477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0459f6590484248829354aa6d479318: Network error: Client connection negotiation failed: client connection to 127.27.174.65:44477: connect: Connection refused (error 111)
I20240502 14:09:06.501497 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7762
I20240502 14:09:06.525621 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7369
I20240502 14:09:06.555935 28345 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240502 14:09:06.893427 28345 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240502 14:09:06.893800 28345 mini_ranger.cc:66] Stopping Ranger...
I20240502 14:09:07.240173 28345 mini_ranger.cc:68] Stopped Ranger
2024-05-02 14:09:07.240 UTC [6416] LOG: received smart shutdown request
2024-05-02 14:09:07.244 UTC [6416] LOG: background worker "logical replication launcher" (PID 6425) exited with exit code 1
2024-05-02 14:09:07.244 UTC [6420] LOG: shutting down
2024-05-02 14:09:07.403 UTC [6416] LOG: database system is shut down
2024-05-02T14:09:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59741 ms)
[ RUN ] SecurityITest.IPKICACert
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1045: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.IPKICACert (6 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:09:07 dist-test-slave-dlb8 krb5kdc[7952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:09:07 dist-test-slave-dlb8 krb5kdc[7952](info): set up 2 sockets
May 02 14:09:07 dist-test-slave-dlb8 krb5kdc[7952](info): commencing operation
krb5kdc: starting...
W20240502 14:09:08.517122 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:09:08 dist-test-slave-dlb8 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:09:08.731667 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45279
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:09.012277 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:09.012857 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:09.013439 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:09.041363 7965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:09.041710 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:09.042040 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:09.042310 7965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:09.042542 7965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:09.075414 7965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:09.076560 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:09.077558 7965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:09.092073 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:09.092926 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:09.094120 7965 server_base.cc:1034] running on GCE node
W20240502 14:09:09.093840 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:10.126986 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:10.130138 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:10.131613 7965 hybrid_clock.cc:648] HybridClock initialized: now 1714658950131574 us; error 51 us; skew 500 ppm
I20240502 14:09:10.132403 7965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:10 dist-test-slave-dlb8 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:10.138453 7965 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:09:10.143615 7965 webserver.cc:460] Webserver started at http://127.27.174.126:33159/ using document root <none> and password file <none>
I20240502 14:09:10.144524 7965 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:10.144742 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:10.145176 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:10.150590 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "e22336448a584ec785bf6a9c6ffe0cfc"
format_stamp: "Formatted at 2024-05-02 14:09:10 on dist-test-slave-dlb8"
server_key: "415f0e69f1c7bb1862de37656364deb3"
server_key_iv: "9a0eba10cd4940d4ab2da0701e82e69b"
server_key_version: "encryptionkey@0"
I20240502 14:09:10.151757 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "e22336448a584ec785bf6a9c6ffe0cfc"
format_stamp: "Formatted at 2024-05-02 14:09:10 on dist-test-slave-dlb8"
server_key: "415f0e69f1c7bb1862de37656364deb3"
server_key_iv: "9a0eba10cd4940d4ab2da0701e82e69b"
server_key_version: "encryptionkey@0"
I20240502 14:09:10.159271 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20240502 14:09:10.164779 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:10.165778 7965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 14:09:10.166129 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "e22336448a584ec785bf6a9c6ffe0cfc"
format_stamp: "Formatted at 2024-05-02 14:09:10 on dist-test-slave-dlb8"
server_key: "415f0e69f1c7bb1862de37656364deb3"
server_key_iv: "9a0eba10cd4940d4ab2da0701e82e69b"
server_key_version: "encryptionkey@0"
I20240502 14:09:10.166503 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:10.232051 7965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:10.237303 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:10.237761 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:10.304044 7965 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45279
I20240502 14:09:10.304121 8033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45279 every 8 connection(s)
I20240502 14:09:10.306584 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:10.308467 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 7965
I20240502 14:09:10.309003 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:10.310606 28345 external_mini_cluster.cc:1428] Setting key 6b752443dbed913248f41d4f494ef499
I20240502 14:09:10.312532 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:09:10 dist-test-slave-dlb8 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:10.339077 8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc: Bootstrap starting.
I20240502 14:09:10.343866 8034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:10.345667 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:10.313151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.27.174.126:45279)
0502 14:09:10.315998 (+ 2847us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:10.316022 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:10.316068 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:09:10.321927 (+ 5859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:10.321967 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:10.322352 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:10.322690 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:10.326508 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:10.327961 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:10.329670 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:10.330064 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:10.335518 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:10.335611 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:10.335639 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:10.335738 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:10.339734 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:10.340683 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:10.340704 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:10.340732 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:10.340896 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:10.341398 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:10.341416 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:10.341429 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:10.342003 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:10.342472 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:10.343082 (+ 610us) server_negotiation.cc:299] Negotiation successful
0502 14:09:10.343683 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2330,"thread_start_us":1857,"threads_started":1}
I20240502 14:09:10.346029 8034 log.cc:824] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:10.349689 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:09:10.351013 8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc: No bootstrap required, opened a new log
I20240502 14:09:10.370172 8034 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } }
I20240502 14:09:10.370901 8034 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:10.371106 8034 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22336448a584ec785bf6a9c6ffe0cfc, State: Initialized, Role: FOLLOWER
I20240502 14:09:10.371770 8034 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } }
I20240502 14:09:10.372344 8034 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:10.372661 8034 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:10.372980 8034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:10.377121 8034 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } }
I20240502 14:09:10.377830 8034 leader_election.cc:302] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22336448a584ec785bf6a9c6ffe0cfc; no voters:
I20240502 14:09:10.379498 8034 leader_election.cc:288] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:10.379863 8039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:10.382007 8039 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [term 1 LEADER]: Becoming Leader. State: Replica: e22336448a584ec785bf6a9c6ffe0cfc, State: Running, Role: LEADER
I20240502 14:09:10.382887 8039 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } }
I20240502 14:09:10.383797 8034 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:10.392809 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } } }
I20240502 14:09:10.393882 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:10.394692 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22336448a584ec785bf6a9c6ffe0cfc. Latest consensus state: current_term: 1 leader_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22336448a584ec785bf6a9c6ffe0cfc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45279 } } }
I20240502 14:09:10.395393 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:10.396950 8048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:10.408314 8048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:10.428406 8048 catalog_manager.cc:1347] Generated new cluster ID: 983c8e93757042ffb3aa739f4b8c6cd8
I20240502 14:09:10.428670 8048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:10.440945 8048 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:10.442351 8048 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:10.456748 8048 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e22336448a584ec785bf6a9c6ffe0cfc: Generated new TSK 0
I20240502 14:09:10.457577 8048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:09:10.474834 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 7965
2024-05-02T14:09:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2922 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1192: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (6 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1219: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedTSK (6 ms)
[----------] 30 tests from SecurityITest (293370 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:09:10 dist-test-slave-dlb8 krb5kdc[8061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:09:10 dist-test-slave-dlb8 krb5kdc[8061](info): set up 2 sockets
May 02 14:09:10 dist-test-slave-dlb8 krb5kdc[8061](info): commencing operation
krb5kdc: starting...
W20240502 14:09:11.464603 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:09:11 dist-test-slave-dlb8 krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:09:11.676836 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33471
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:11.954226 8074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:11.954820 8074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:11.955467 8074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:11.983536 8074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:11.983915 8074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:11.984198 8074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:11.984434 8074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:11.984666 8074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:12.017980 8074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:12.019097 8074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:12.020054 8074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:12.036206 8080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:12.036726 8083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:12.037853 8074 server_base.cc:1034] running on GCE node
W20240502 14:09:12.037112 8081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:13.072921 8074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:13.075973 8074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:13.077414 8074 hybrid_clock.cc:648] HybridClock initialized: now 1714658953077377 us; error 59 us; skew 500 ppm
I20240502 14:09:13.078171 8074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:13 dist-test-slave-dlb8 krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:13.088754 8074 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:09:13.093555 8074 webserver.cc:460] Webserver started at http://127.27.174.126:42667/ using document root <none> and password file <none>
I20240502 14:09:13.094496 8074 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:13.094699 8074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:13.095185 8074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:13.099964 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "148d3552277341b5a227771aa4359939"
format_stamp: "Formatted at 2024-05-02 14:09:13 on dist-test-slave-dlb8"
server_key: "42dcacc92b29f4b817ea2e50c3946e02"
server_key_iv: "c99e7e1ca1cf7dc8fcf44478928384ba"
server_key_version: "encryptionkey@0"
I20240502 14:09:13.101151 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "148d3552277341b5a227771aa4359939"
format_stamp: "Formatted at 2024-05-02 14:09:13 on dist-test-slave-dlb8"
server_key: "42dcacc92b29f4b817ea2e50c3946e02"
server_key_iv: "c99e7e1ca1cf7dc8fcf44478928384ba"
server_key_version: "encryptionkey@0"
I20240502 14:09:13.108129 8074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20240502 14:09:13.113319 8091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:13.114360 8074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:09:13.114673 8074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "148d3552277341b5a227771aa4359939"
format_stamp: "Formatted at 2024-05-02 14:09:13 on dist-test-slave-dlb8"
server_key: "42dcacc92b29f4b817ea2e50c3946e02"
server_key_iv: "c99e7e1ca1cf7dc8fcf44478928384ba"
server_key_version: "encryptionkey@0"
I20240502 14:09:13.115042 8074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:13.169199 8074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:13.174427 8074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:13.174891 8074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:13.237618 8074 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33471
I20240502 14:09:13.237694 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33471 every 8 connection(s)
I20240502 14:09:13.240082 8074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:13.244921 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:13.250255 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8074
I20240502 14:09:13.250658 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:13.251664 28345 external_mini_cluster.cc:1428] Setting key 68f686e30103de923dc0047ae9be4428
I20240502 14:09:13.268291 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: Bootstrap starting.
May 02 14:09:13 dist-test-slave-dlb8 krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:13.274629 8143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:13.276310 8143 log.cc:824] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:13.281625 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: No bootstrap required, opened a new log
I20240502 14:09:13.284412 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:13.254183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.27.174.126:33471)
0502 14:09:13.256802 (+ 2619us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:13.256827 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:13.256874 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:09:13.260392 (+ 3518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:13.260434 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:13.260757 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:13.261060 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:13.264350 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:13.265808 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:13.267660 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:13.268077 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:13.273857 (+ 5780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:13.273957 (+ 100us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:13.273988 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:13.274124 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:13.278628 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:13.279598 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:13.279632 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:13.279678 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:13.279857 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:13.280427 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:13.280444 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:13.280456 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:13.280964 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:13.281280 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:13.282004 (+ 724us) server_negotiation.cc:299] Negotiation successful
0502 14:09:13.282459 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":1685,"threads_started":1}
I20240502 14:09:13.301333 8143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } }
I20240502 14:09:13.301998 8143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:13.302279 8143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148d3552277341b5a227771aa4359939, State: Initialized, Role: FOLLOWER
I20240502 14:09:13.302981 8143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } }
I20240502 14:09:13.303440 8143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:13.303644 8143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:13.303911 8143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:13.307793 8143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } }
I20240502 14:09:13.308388 8143 leader_election.cc:302] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 148d3552277341b5a227771aa4359939; no voters:
I20240502 14:09:13.310310 8143 leader_election.cc:288] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:13.310655 8148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:13.313308 8148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [term 1 LEADER]: Becoming Leader. State: Replica: 148d3552277341b5a227771aa4359939, State: Running, Role: LEADER
I20240502 14:09:13.314134 8148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } }
I20240502 14:09:13.315013 8143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:13.325212 8149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "148d3552277341b5a227771aa4359939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } } }
I20240502 14:09:13.325327 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 148d3552277341b5a227771aa4359939. Latest consensus state: current_term: 1 leader_uuid: "148d3552277341b5a227771aa4359939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148d3552277341b5a227771aa4359939" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33471 } } }
I20240502 14:09:13.326159 8149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:13.326161 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:13.330899 8157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:13.342511 8157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:13.360134 8157 catalog_manager.cc:1347] Generated new cluster ID: d1f18dc00dfd4511bf6f9bc2bffa9329
I20240502 14:09:13.360424 8157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:13.374452 8157 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:13.375857 8157 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:13.390406 8157 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: Generated new TSK 0
I20240502 14:09:13.391297 8157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:09:13.444723 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:13.737736 8169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:13.738282 8169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:13.738904 8169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:13.769026 8169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:13.769381 8169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:13.770242 8169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:13.803830 8169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:13.804977 8169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:13.805943 8169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:13.822819 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:13.824213 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:13.825286 8169 server_base.cc:1034] running on GCE node
W20240502 14:09:13.823508 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:14.864202 8169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:14.874547 8169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:14.875923 8169 hybrid_clock.cc:648] HybridClock initialized: now 1714658954875898 us; error 44 us; skew 500 ppm
I20240502 14:09:14.876667 8169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:14 dist-test-slave-dlb8 krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658954, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:14.882535 8169 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:09:14.887821 8169 webserver.cc:460] Webserver started at http://127.27.174.65:38577/ using document root <none> and password file <none>
I20240502 14:09:14.888872 8169 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:14.889097 8169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:14.889580 8169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:14.896242 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "35c27a910bca42c88f03da5d74cf2a62"
format_stamp: "Formatted at 2024-05-02 14:09:14 on dist-test-slave-dlb8"
server_key: "72db71c27ef409ba468c599967e2717d"
server_key_iv: "ddf97aada5c63b729a6f9b448250cee2"
server_key_version: "encryptionkey@0"
I20240502 14:09:14.897645 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "35c27a910bca42c88f03da5d74cf2a62"
format_stamp: "Formatted at 2024-05-02 14:09:14 on dist-test-slave-dlb8"
server_key: "72db71c27ef409ba468c599967e2717d"
server_key_iv: "ddf97aada5c63b729a6f9b448250cee2"
server_key_version: "encryptionkey@0"
I20240502 14:09:14.906165 8169 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20240502 14:09:14.913023 8186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:14.914153 8169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:09:14.914523 8169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "35c27a910bca42c88f03da5d74cf2a62"
format_stamp: "Formatted at 2024-05-02 14:09:14 on dist-test-slave-dlb8"
server_key: "72db71c27ef409ba468c599967e2717d"
server_key_iv: "ddf97aada5c63b729a6f9b448250cee2"
server_key_version: "encryptionkey@0"
I20240502 14:09:14.915016 8169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:14.985116 8169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:14.991925 8169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:14.992482 8169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:14.995782 8169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:15.001013 8169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:15.001346 8169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:15.001672 8169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:15.001922 8169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:15.137598 8169 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:35923
I20240502 14:09:15.137696 8299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:35923 every 8 connection(s)
I20240502 14:09:15.140235 8169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:15.149082 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8169
I20240502 14:09:15.149677 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:15.151429 28345 external_mini_cluster.cc:1428] Setting key 58f15be854de23906ca673b34dc85b57
May 02 14:09:15 dist-test-slave-dlb8 krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658954, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:15.175016 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:15.144298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:54251 (local address 127.27.174.126:33471)
0502 14:09:15.145500 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:15.145515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:15.147976 (+ 2461us) server_negotiation.cc:407] Connection header received
0502 14:09:15.154594 (+ 6618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:15.154614 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:15.154776 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:15.154946 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:15.157421 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:15.159260 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:15.160618 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:15.161259 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:15.164989 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:15.165061 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:15.165079 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:15.165178 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:15.168763 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:15.170309 (+ 1546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:15.170332 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:15.170346 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:15.170481 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:15.171276 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:15.171301 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:15.171315 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:15.171704 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:15.171887 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:15.173236 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0502 14:09:15.173597 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":514,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20240502 14:09:15.176635 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:15.144645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33471 (local address 127.27.174.65:54251)
0502 14:09:15.147779 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:15.147843 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:15.154236 (+ 6393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:15.155257 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:15.155299 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:15.156140 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:15.157103 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:15.157142 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:15.159495 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:15.159523 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:15.160365 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:15.160403 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:15.160692 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:15.161542 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:15.161608 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:15.164697 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:15.168986 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:15.169026 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:15.169070 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:15.170128 (+ 1058us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:15.170673 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:15.170717 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:15.170751 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:15.171056 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:15.171943 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:15.171988 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:15.172629 (+ 641us) client_negotiation.cc:769] Sending connection context
0502 14:09:15.173120 (+ 491us) client_negotiation.cc:240] Negotiation successful
0502 14:09:15.173846 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"thread_start_us":1788,"threads_started":1}
I20240502 14:09:15.178495 8300 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33471
I20240502 14:09:15.178853 8300 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:09:15.179847 8300 heartbeater.cc:511] Master 127.27.174.126:33471 requested a full tablet report, sending...
I20240502 14:09:15.182313 8108 ts_manager.cc:194] Registered new tserver with Master: 35c27a910bca42c88f03da5d74cf2a62 (127.27.174.65:35923)
I20240502 14:09:15.184082 8108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:54251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:09:15.204515 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:15.499441 8306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:15.499889 8306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:15.500424 8306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:15.528939 8306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:15.529240 8306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:15.530010 8306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:15.562690 8306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:15.563755 8306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:15.564815 8306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:15.579921 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:16.187342 8300 heartbeater.cc:503] Master 127.27.174.126:33471 was elected leader, sending a full tablet report...
W20240502 14:09:15.580222 8313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:15.581939 8306 server_base.cc:1034] running on GCE node
W20240502 14:09:15.582468 8315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:16.605551 8306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:16.607826 8306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:16.609159 8306 hybrid_clock.cc:648] HybridClock initialized: now 1714658956609135 us; error 43 us; skew 500 ppm
I20240502 14:09:16.609890 8306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:16 dist-test-slave-dlb8 krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:16.615650 8306 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:09:16.620054 8306 webserver.cc:460] Webserver started at http://127.27.174.66:34815/ using document root <none> and password file <none>
I20240502 14:09:16.620889 8306 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:16.621062 8306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:16.621433 8306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:16.626499 8306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "50915c64c56140a0a7f756166ada6fff"
format_stamp: "Formatted at 2024-05-02 14:09:16 on dist-test-slave-dlb8"
server_key: "52e2e986aecfdb966a24c1a1016819c3"
server_key_iv: "cad0a8c5811542bd2b0ee254c082da9d"
server_key_version: "encryptionkey@0"
I20240502 14:09:16.627560 8306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "50915c64c56140a0a7f756166ada6fff"
format_stamp: "Formatted at 2024-05-02 14:09:16 on dist-test-slave-dlb8"
server_key: "52e2e986aecfdb966a24c1a1016819c3"
server_key_iv: "cad0a8c5811542bd2b0ee254c082da9d"
server_key_version: "encryptionkey@0"
I20240502 14:09:16.634486 8306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:09:16.639811 8323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:16.640823 8306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:09:16.641122 8306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "50915c64c56140a0a7f756166ada6fff"
format_stamp: "Formatted at 2024-05-02 14:09:16 on dist-test-slave-dlb8"
server_key: "52e2e986aecfdb966a24c1a1016819c3"
server_key_iv: "cad0a8c5811542bd2b0ee254c082da9d"
server_key_version: "encryptionkey@0"
I20240502 14:09:16.641475 8306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:16.686229 8306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:16.693132 8306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:16.693542 8306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:16.696012 8306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:16.699626 8306 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:16.699795 8306 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:16.700024 8306 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:16.700145 8306 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:16.830396 8306 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:38385
I20240502 14:09:16.830498 8435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:38385 every 8 connection(s)
I20240502 14:09:16.832950 8306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:16.839793 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8306
I20240502 14:09:16.840222 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:16.841305 28345 external_mini_cluster.cc:1428] Setting key 78c8c3ac84e5f1bc400eeb8b2b4233e9
May 02 14:09:16 dist-test-slave-dlb8 krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:09:16.864908 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:16.837136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33471 (local address 127.27.174.66:52317)
0502 14:09:16.839486 (+ 2350us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:16.839550 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:16.844308 (+ 4758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:16.845327 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:16.845352 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:16.846199 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:16.847153 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:16.847195 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:16.849156 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:16.849179 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:16.850029 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:16.850109 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:16.850389 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:16.851206 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:16.851275 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:16.854858 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:16.859117 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:16.859140 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:16.859162 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:16.859787 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:16.860507 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:16.860525 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:16.860539 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:16.860733 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:16.861860 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:16.861882 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:16.862292 (+ 410us) client_negotiation.cc:769] Sending connection context
0502 14:09:16.862586 (+ 294us) client_negotiation.cc:240] Negotiation successful
0502 14:09:16.862999 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"thread_start_us":1390,"threads_started":1}
I20240502 14:09:16.865317 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:16.836912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:52317 (local address 127.27.174.126:33471)
0502 14:09:16.838751 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:16.838773 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:16.839712 (+ 939us) server_negotiation.cc:407] Connection header received
0502 14:09:16.844660 (+ 4948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:16.844686 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:16.844913 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:16.845097 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:16.847474 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:16.848916 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:16.850302 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:16.850924 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:16.855157 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:16.855231 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:16.855250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:16.855360 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:16.858904 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:16.860046 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:16.860090 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:16.860121 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:16.860327 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:16.860940 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:16.860980 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:16.861012 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:16.861654 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:16.861952 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:16.862884 (+ 932us) server_negotiation.cc:299] Negotiation successful
0502 14:09:16.863409 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":521,"threads_started":1}
I20240502 14:09:16.866964 8436 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:09:16.867415 8436 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:16.868741 8436 heartbeater.cc:511] Master 127.27.174.126:33471 requested a full tablet report, sending...
I20240502 14:09:16.871235 8108 ts_manager.cc:194] Registered new tserver with Master: 50915c64c56140a0a7f756166ada6fff (127.27.174.66:38385)
I20240502 14:09:16.872421 8108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:52317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:09:16.893573 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:17.175334 8442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:17.175809 8442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:17.176349 8442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:17.204540 8442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:17.204905 8442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:17.205785 8442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:17.238919 8442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:17.240010 8442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:17.240976 8442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:17.258704 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:17.875599 8436 heartbeater.cc:503] Master 127.27.174.126:33471 was elected leader, sending a full tablet report...
W20240502 14:09:17.259092 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:17.259500 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:17.259963 8442 server_base.cc:1034] running on GCE node
I20240502 14:09:18.272100 8442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:18.274408 8442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:18.275748 8442 hybrid_clock.cc:648] HybridClock initialized: now 1714658958275711 us; error 57 us; skew 500 ppm
I20240502 14:09:18.276474 8442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:18 dist-test-slave-dlb8 krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658958, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:18.287376 8442 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:09:18.292241 8442 webserver.cc:460] Webserver started at http://127.27.174.67:36361/ using document root <none> and password file <none>
I20240502 14:09:18.293133 8442 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:18.293334 8442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:18.293736 8442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:18.298805 8442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "ad3803e6ff724817a30e0b26639aaad8"
format_stamp: "Formatted at 2024-05-02 14:09:18 on dist-test-slave-dlb8"
server_key: "6821e394f04167d753024f8c84afbc1f"
server_key_iv: "d1c9db1254fbbcd4786c67e93515b684"
server_key_version: "encryptionkey@0"
I20240502 14:09:18.299885 8442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "ad3803e6ff724817a30e0b26639aaad8"
format_stamp: "Formatted at 2024-05-02 14:09:18 on dist-test-slave-dlb8"
server_key: "6821e394f04167d753024f8c84afbc1f"
server_key_iv: "d1c9db1254fbbcd4786c67e93515b684"
server_key_version: "encryptionkey@0"
I20240502 14:09:18.306720 8442 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240502 14:09:18.311677 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:18.312608 8442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:09:18.312881 8442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "ad3803e6ff724817a30e0b26639aaad8"
format_stamp: "Formatted at 2024-05-02 14:09:18 on dist-test-slave-dlb8"
server_key: "6821e394f04167d753024f8c84afbc1f"
server_key_iv: "d1c9db1254fbbcd4786c67e93515b684"
server_key_version: "encryptionkey@0"
I20240502 14:09:18.313217 8442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:18.380978 8442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:18.385874 8442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:18.386296 8442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:18.389063 8442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:18.393505 8442 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:18.393695 8442 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:18.393878 8442 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:18.393999 8442 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:18.519927 8442 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:40345
I20240502 14:09:18.520025 8572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:40345 every 8 connection(s)
I20240502 14:09:18.522439 8442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:18.527585 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8442
I20240502 14:09:18.528348 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:18.529490 28345 external_mini_cluster.cc:1428] Setting key 420bc9beda6b4dfd792865a6ae859635
May 02 14:09:18 dist-test-slave-dlb8 krb5kdc[8061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658958, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:18.554267 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.526433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:44065 (local address 127.27.174.126:33471)
0502 14:09:18.528020 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.528053 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.530016 (+ 1963us) server_negotiation.cc:407] Connection header received
0502 14:09:18.535785 (+ 5769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.535804 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.535972 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.536117 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:18.538284 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.539246 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.540477 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.540835 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.545029 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:18.545107 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:18.545125 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:18.545226 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:18.549567 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:18.550553 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:18.550575 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:18.550590 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:18.550716 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:18.551178 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:18.551194 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:18.551206 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:18.551561 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:18.551728 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.552544 (+ 816us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.552905 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":604,"threads_started":1}
I20240502 14:09:18.554838 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.526728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33471 (local address 127.27.174.67:44065)
0502 14:09:18.529772 (+ 3044us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.529839 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.535493 (+ 5654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.536329 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.536355 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.537088 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:18.538033 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.538109 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.539438 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.539459 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.540275 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.540314 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.540599 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.541414 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:18.541473 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:18.544778 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:18.549744 (+ 4966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:18.549766 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:18.549787 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:18.550400 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:18.550840 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:18.550859 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:18.550872 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:18.551049 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:18.551697 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:18.551714 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:18.552051 (+ 337us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.552372 (+ 321us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.552790 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2330,"thread_start_us":1677,"threads_started":1}
I20240502 14:09:18.556519 8573 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33471
I20240502 14:09:18.556860 8573 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:18.557781 8573 heartbeater.cc:511] Master 127.27.174.126:33471 requested a full tablet report, sending...
I20240502 14:09:18.559784 8108 ts_manager.cc:194] Registered new tserver with Master: ad3803e6ff724817a30e0b26639aaad8 (127.27.174.67:40345)
I20240502 14:09:18.561080 8108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:44065
I20240502 14:09:18.563062 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:18.596088 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.577866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.27.174.126:33471)
0502 14:09:18.578393 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.578420 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.579417 (+ 997us) server_negotiation.cc:407] Connection header received
0502 14:09:18.579798 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.579817 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.579984 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.580126 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:18.582565 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.583362 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.584726 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.585269 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.587637 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:18.587735 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:18.587761 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:18.587891 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:18.591280 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:18.592208 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:18.592227 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:18.592242 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:18.592378 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:18.592879 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:18.592897 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:18.592910 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:18.593307 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:18.593655 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.594246 (+ 591us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.594728 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20240502 14:09:18.601944 8107 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:18.621450 8107 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:18.668044 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.648899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38385 (local address 127.0.0.1:42454)
0502 14:09:18.653084 (+ 4185us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.653186 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.653510 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.659748 (+ 6238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.659780 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.659886 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:18.660657 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.660714 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.663403 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.663434 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.665434 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.665499 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.665834 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.665973 (+ 139us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.666171 (+ 198us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.666427 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3569,"thread_start_us":2809,"threads_started":1}
I20240502 14:09:18.669144 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.650345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.27.174.66:38385)
0502 14:09:18.657694 (+ 7349us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.657731 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.657789 (+ 58us) server_negotiation.cc:407] Connection header received
0502 14:09:18.657990 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.658037 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.658409 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.658641 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:18.660954 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.663133 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.665734 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.667032 (+ 1298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.667285 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.667504 (+ 219us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.667743 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3430,"thread_start_us":3250,"threads_started":1}
I20240502 14:09:18.671830 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.653309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:40345 (local address 127.0.0.1:35480)
0502 14:09:18.658768 (+ 5459us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.658993 (+ 225us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.659216 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.661916 (+ 2700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.661948 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.662072 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:18.662948 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.663008 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.666141 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.666175 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.668053 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.668110 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.670188 (+ 2078us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.670270 (+ 82us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.670435 (+ 165us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.670662 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4853,"thread_start_us":4000,"threads_started":1}
I20240502 14:09:18.673765 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.647873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.27.174.65:35923)
0502 14:09:18.660112 (+ 12239us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.660898 (+ 786us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.660958 (+ 60us) server_negotiation.cc:407] Connection header received
0502 14:09:18.661137 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.661167 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.661484 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.661689 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:18.663687 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.665121 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.670021 (+ 4900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.671197 (+ 1176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.671425 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.672455 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.672669 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11604,"thread_start_us":11304,"threads_started":1}
I20240502 14:09:18.673849 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.647409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35923 (local address 127.0.0.1:60140)
0502 14:09:18.659999 (+ 12590us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.660059 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.660298 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.662229 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.662258 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.662360 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:18.663335 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.663386 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.665372 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.665408 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.667282 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.667342 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.671615 (+ 4273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.671713 (+ 98us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.671878 (+ 165us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.672175 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4703,"thread_start_us":2703,"threads_started":1}
I20240502 14:09:18.673885 8371 tablet_service.cc:1467] Processing CreateTablet for tablet 97384da25f6f4416ba265e574b639a9f (DEFAULT_TABLE table=test-table [id=9e2bc651624c4122a93f9c9558e7155d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:18.676338 8371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97384da25f6f4416ba265e574b639a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:18.677682 8234 tablet_service.cc:1467] Processing CreateTablet for tablet 97384da25f6f4416ba265e574b639a9f (DEFAULT_TABLE table=test-table [id=9e2bc651624c4122a93f9c9558e7155d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:18.678124 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.654980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.27.174.67:40345)
0502 14:09:18.660576 (+ 5596us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.660596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.660633 (+ 37us) server_negotiation.cc:407] Connection header received
0502 14:09:18.660801 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.660833 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.661182 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.661391 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:18.663402 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.665832 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.668398 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.669679 (+ 1281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.669918 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.674821 (+ 4903us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.675080 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4947,"thread_start_us":4525,"threads_started":1}
I20240502 14:09:18.679775 8234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97384da25f6f4416ba265e574b639a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:18.681972 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 97384da25f6f4416ba265e574b639a9f (DEFAULT_TABLE table=test-table [id=9e2bc651624c4122a93f9c9558e7155d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:18.683769 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97384da25f6f4416ba265e574b639a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:18.704223 8593 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: Bootstrap starting.
I20240502 14:09:18.707779 8592 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Bootstrap starting.
I20240502 14:09:18.711004 8593 tablet_bootstrap.cc:654] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:18.712380 8594 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Bootstrap starting.
I20240502 14:09:18.713920 8593 log.cc:824] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:18.715204 8592 tablet_bootstrap.cc:654] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:18.717548 8592 log.cc:824] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:18.719139 8594 tablet_bootstrap.cc:654] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:18.720721 8593 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: No bootstrap required, opened a new log
I20240502 14:09:18.721138 8593 ts_tablet_manager.cc:1367] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20240502 14:09:18.721495 8594 log.cc:824] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:18.723531 8592 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: No bootstrap required, opened a new log
I20240502 14:09:18.724066 8592 ts_tablet_manager.cc:1367] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 14:09:18.726827 8594 tablet_bootstrap.cc:492] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: No bootstrap required, opened a new log
I20240502 14:09:18.727295 8594 ts_tablet_manager.cc:1367] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240502 14:09:18.749946 8594 raft_consensus.cc:348] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.750703 8594 raft_consensus.cc:374] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:18.750969 8594 raft_consensus.cc:729] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3803e6ff724817a30e0b26639aaad8, State: Initialized, Role: FOLLOWER
I20240502 14:09:18.751765 8594 consensus_queue.cc:260] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.753603 8593 raft_consensus.cc:348] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.754644 8593 raft_consensus.cc:374] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:18.755007 8593 raft_consensus.cc:729] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c27a910bca42c88f03da5d74cf2a62, State: Initialized, Role: FOLLOWER
I20240502 14:09:18.755177 8573 heartbeater.cc:503] Master 127.27.174.126:33471 was elected leader, sending a full tablet report...
I20240502 14:09:18.755841 8594 ts_tablet_manager.cc:1398] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Time spent starting tablet: real 0.028s user 0.012s sys 0.015s
I20240502 14:09:18.756227 8598 raft_consensus.cc:482] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:18.755949 8593 consensus_queue.cc:260] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.757617 8592 raft_consensus.cc:348] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.758554 8592 raft_consensus.cc:374] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:18.758881 8592 raft_consensus.cc:729] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50915c64c56140a0a7f756166ada6fff, State: Initialized, Role: FOLLOWER
I20240502 14:09:18.759802 8592 consensus_queue.cc:260] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.760385 8593 ts_tablet_manager.cc:1398] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62: Time spent starting tablet: real 0.039s user 0.026s sys 0.015s
I20240502 14:09:18.756763 8598 raft_consensus.cc:504] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.764221 8598 leader_election.cc:288] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35c27a910bca42c88f03da5d74cf2a62 (127.27.174.65:35923), 50915c64c56140a0a7f756166ada6fff (127.27.174.66:38385)
I20240502 14:09:18.765550 8592 ts_tablet_manager.cc:1398] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Time spent starting tablet: real 0.041s user 0.027s sys 0.011s
I20240502 14:09:18.774937 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.764468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35923 (local address 127.27.174.67:33437)
0502 14:09:18.765267 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.765319 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.765563 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.767107 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.767136 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.767235 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:18.767621 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.767654 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.771090 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.771125 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.772685 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.772741 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.773077 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.773164 (+ 87us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.773296 (+ 132us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.773486 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
W20240502 14:09:18.777035 8574 tablet.cc:2369] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:18.780292 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.765201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:33437 (local address 127.27.174.65:35923)
0502 14:09:18.765932 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.765961 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.766006 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:09:18.766318 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.766342 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.766595 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.766749 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:18.767874 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.769094 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.776514 (+ 7420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.777703 (+ 1189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.777854 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.778056 (+ 202us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.778286 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20240502 14:09:18.780473 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.768431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:37431 (local address 127.27.174.66:38385)
0502 14:09:18.769487 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.769514 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.769865 (+ 351us) server_negotiation.cc:407] Connection header received
0502 14:09:18.771208 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.771238 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.771483 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.771612 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:18.773448 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.774703 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.777156 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.778264 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.778399 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.779008 (+ 609us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.779210 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508}
I20240502 14:09:18.781422 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.765478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:38385 (local address 127.27.174.67:37431)
0502 14:09:18.769519 (+ 4041us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.769568 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:18.770097 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:18.771993 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:18.772019 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:18.772110 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:18.773156 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.773205 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.775065 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.775099 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:18.776828 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:18.776875 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.778637 (+ 1762us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.778725 (+ 88us) client_negotiation.cc:769] Sending connection context
0502 14:09:18.779577 (+ 852us) client_negotiation.cc:240] Negotiation successful
0502 14:09:18.779765 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3407,"thread_start_us":3095,"threads_started":1}
I20240502 14:09:18.781980 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97384da25f6f4416ba265e574b639a9f" candidate_uuid: "ad3803e6ff724817a30e0b26639aaad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c27a910bca42c88f03da5d74cf2a62" is_pre_election: true
I20240502 14:09:18.782831 8254 raft_consensus.cc:2454] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3803e6ff724817a30e0b26639aaad8 in term 0.
I20240502 14:09:18.783334 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97384da25f6f4416ba265e574b639a9f" candidate_uuid: "ad3803e6ff724817a30e0b26639aaad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50915c64c56140a0a7f756166ada6fff" is_pre_election: true
I20240502 14:09:18.784041 8391 raft_consensus.cc:2454] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3803e6ff724817a30e0b26639aaad8 in term 0.
I20240502 14:09:18.784024 8461 leader_election.cc:302] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c27a910bca42c88f03da5d74cf2a62, ad3803e6ff724817a30e0b26639aaad8; no voters:
I20240502 14:09:18.784734 8598 raft_consensus.cc:2781] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:18.784977 8598 raft_consensus.cc:482] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:18.785226 8598 raft_consensus.cc:3037] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:18.789880 8598 raft_consensus.cc:504] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.791180 8598 leader_election.cc:288] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [CANDIDATE]: Term 1 election: Requested vote from peers 35c27a910bca42c88f03da5d74cf2a62 (127.27.174.65:35923), 50915c64c56140a0a7f756166ada6fff (127.27.174.66:38385)
I20240502 14:09:18.791944 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97384da25f6f4416ba265e574b639a9f" candidate_uuid: "ad3803e6ff724817a30e0b26639aaad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c27a910bca42c88f03da5d74cf2a62"
I20240502 14:09:18.792063 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97384da25f6f4416ba265e574b639a9f" candidate_uuid: "ad3803e6ff724817a30e0b26639aaad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50915c64c56140a0a7f756166ada6fff"
I20240502 14:09:18.792433 8254 raft_consensus.cc:3037] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:18.792529 8391 raft_consensus.cc:3037] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:18.799405 8391 raft_consensus.cc:2454] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad3803e6ff724817a30e0b26639aaad8 in term 1.
I20240502 14:09:18.799409 8254 raft_consensus.cc:2454] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad3803e6ff724817a30e0b26639aaad8 in term 1.
I20240502 14:09:18.800487 8464 leader_election.cc:302] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50915c64c56140a0a7f756166ada6fff, ad3803e6ff724817a30e0b26639aaad8; no voters:
I20240502 14:09:18.801157 8598 raft_consensus.cc:2781] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:18.802664 8598 raft_consensus.cc:686] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 1 LEADER]: Becoming Leader. State: Replica: ad3803e6ff724817a30e0b26639aaad8, State: Running, Role: LEADER
I20240502 14:09:18.803345 8598 consensus_queue.cc:237] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } }
I20240502 14:09:18.814556 8106 catalog_manager.cc:5484] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad3803e6ff724817a30e0b26639aaad8 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "ad3803e6ff724817a30e0b26639aaad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad3803e6ff724817a30e0b26639aaad8" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 40345 } health_report { overall_health: HEALTHY } } }
W20240502 14:09:18.839402 8437 tablet.cc:2369] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:18.846848 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.839162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.27.174.67:40345)
0502 14:09:18.839890 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:18.839915 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:18.840197 (+ 282us) server_negotiation.cc:407] Connection header received
0502 14:09:18.840603 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:18.840620 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:18.840828 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:18.840947 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:18.841853 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.842660 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:18.843976 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:18.844547 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:18.844752 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:18.845325 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:18.845530 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:18.845755 (+ 225us) server_negotiation.cc:299] Negotiation successful
0502 14:09:18.845894 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20240502 14:09:18.858949 8254 raft_consensus.cc:1261] T 97384da25f6f4416ba265e574b639a9f P 35c27a910bca42c88f03da5d74cf2a62 [term 1 FOLLOWER]: Refusing update from remote peer ad3803e6ff724817a30e0b26639aaad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:18.858903 8391 raft_consensus.cc:1261] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 1 FOLLOWER]: Refusing update from remote peer ad3803e6ff724817a30e0b26639aaad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:18.860682 8602 consensus_queue.cc:1035] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50915c64c56140a0a7f756166ada6fff" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:09:18.861377 8598 consensus_queue.cc:1035] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c27a910bca42c88f03da5d74cf2a62" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 35923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:18.907514 8106 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:18.908002 8106 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60170:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:18.916225 8106 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 148d3552277341b5a227771aa4359939: Sending DeleteTablet for 3 replicas of tablet 97384da25f6f4416ba265e574b639a9f
I20240502 14:09:18.918195 8371 tablet_service.cc:1514] Processing DeleteTablet for tablet 97384da25f6f4416ba265e574b639a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:42454
I20240502 14:09:18.918231 8234 tablet_service.cc:1514] Processing DeleteTablet for tablet 97384da25f6f4416ba265e574b639a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:60140
I20240502 14:09:18.918336 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 97384da25f6f4416ba265e574b639a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:18 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:35480
I20240502 14:09:18.933261 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8169
I20240502 14:09:18.933326 8617 tablet_replica.cc:330] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: stopping tablet replica
I20240502 14:09:18.935604 8617 raft_consensus.cc:2229] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:18.936887 8617 raft_consensus.cc:2258] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:18.940634 8617 ts_tablet_manager.cc:1875] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:18.946743 8615 tablet_replica.cc:330] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: stopping tablet replica
I20240502 14:09:18.947749 8615 raft_consensus.cc:2229] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:18.948657 8615 raft_consensus.cc:2258] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:18.952127 8615 ts_tablet_manager.cc:1875] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:18.957002 8617 ts_tablet_manager.cc:1888] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:18.957465 8617 log.cc:1196] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/97384da25f6f4416ba265e574b639a9f
I20240502 14:09:18.958351 8617 ts_tablet_manager.cc:1909] T 97384da25f6f4416ba265e574b639a9f P ad3803e6ff724817a30e0b26639aaad8: Deleting consensus metadata
I20240502 14:09:18.961658 8094 catalog_manager.cc:4830] TS ad3803e6ff724817a30e0b26639aaad8 (127.27.174.67:40345): tablet 97384da25f6f4416ba265e574b639a9f (table test-table [id=9e2bc651624c4122a93f9c9558e7155d]) successfully deleted
I20240502 14:09:18.964993 8615 ts_tablet_manager.cc:1888] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:18.965317 8615 log.cc:1196] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/97384da25f6f4416ba265e574b639a9f
I20240502 14:09:18.965919 8615 ts_tablet_manager.cc:1909] T 97384da25f6f4416ba265e574b639a9f P 50915c64c56140a0a7f756166ada6fff: Deleting consensus metadata
I20240502 14:09:18.968247 8095 catalog_manager.cc:4830] TS 50915c64c56140a0a7f756166ada6fff (127.27.174.66:38385): tablet 97384da25f6f4416ba265e574b639a9f (table test-table [id=9e2bc651624c4122a93f9c9558e7155d]) successfully deleted
W20240502 14:09:18.969890 8092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:35923 (error 108)
I20240502 14:09:18.970885 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8306
I20240502 14:09:18.975984 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:18.974494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:35923 (local address 127.0.0.1:60142)
0502 14:09:18.975160 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:18.975416 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20240502 14:09:18.976832 8092 catalog_manager.cc:4573] TS 35c27a910bca42c88f03da5d74cf2a62 (127.27.174.65:35923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97384da25f6f4416ba265e574b639a9f: Network error: Client connection negotiation failed: client connection to 127.27.174.65:35923: connect: Connection refused (error 111)
I20240502 14:09:18.996850 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8442
I20240502 14:09:19.023008 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8074
2024-05-02T14:09:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (8545 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-02T14:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:19Z Disabled control of system clock
I20240502 14:09:19.107744 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41153
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:19.392155 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:19.392876 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:19.393711 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:19.424027 8622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:19.424355 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:19.424609 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:19.424857 8622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:19.425091 8622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:19.458773 8622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:19.459908 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:19.461457 8622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:19.476032 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:19.476635 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:19.477161 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:19.477807 8622 server_base.cc:1034] running on GCE node
I20240502 14:09:20.510008 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:20.512964 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:20.514350 8622 hybrid_clock.cc:648] HybridClock initialized: now 1714658960514323 us; error 46 us; skew 500 ppm
I20240502 14:09:20.515132 8622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:20.521433 8622 webserver.cc:460] Webserver started at http://127.27.174.126:35057/ using document root <none> and password file <none>
I20240502 14:09:20.522368 8622 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:20.522549 8622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:20.522950 8622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:20.527941 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "ec49fb205a1a4dd48eb2a23a86b932b5"
format_stamp: "Formatted at 2024-05-02 14:09:20 on dist-test-slave-dlb8"
server_key: "5337f443b02925636ad5f73c049f79f1"
server_key_iv: "06e713d1f962e6f0b6622f888bee047c"
server_key_version: "encryptionkey@0"
I20240502 14:09:20.529024 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ec49fb205a1a4dd48eb2a23a86b932b5"
format_stamp: "Formatted at 2024-05-02 14:09:20 on dist-test-slave-dlb8"
server_key: "5337f443b02925636ad5f73c049f79f1"
server_key_iv: "06e713d1f962e6f0b6622f888bee047c"
server_key_version: "encryptionkey@0"
I20240502 14:09:20.535750 8622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 14:09:20.540921 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:20.542003 8622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:09:20.542352 8622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "ec49fb205a1a4dd48eb2a23a86b932b5"
format_stamp: "Formatted at 2024-05-02 14:09:20 on dist-test-slave-dlb8"
server_key: "5337f443b02925636ad5f73c049f79f1"
server_key_iv: "06e713d1f962e6f0b6622f888bee047c"
server_key_version: "encryptionkey@0"
I20240502 14:09:20.542692 8622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:20.596779 8622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:20.598135 8622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:20.598515 8622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:20.664216 8622 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41153
I20240502 14:09:20.664315 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41153 every 8 connection(s)
I20240502 14:09:20.666927 8622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:20.672021 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:20.672461 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8622
I20240502 14:09:20.672845 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:20.673866 28345 external_mini_cluster.cc:1428] Setting key 791dde699a030f4940ffdd162eb553db
I20240502 14:09:20.695495 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:20.676069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.27.174.126:41153)
0502 14:09:20.678867 (+ 2798us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:20.678903 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:20.678976 (+ 73us) server_negotiation.cc:407] Connection header received
0502 14:09:20.682948 (+ 3972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:20.683012 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:20.683539 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:20.683988 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:20.685932 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:20.688142 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:20.690160 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:20.691151 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:20.691641 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:20.691755 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:20.691794 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:20.692776 (+ 982us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:20.692898 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:20.692955 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:20.693109 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:20.693532 (+ 423us) server_negotiation.cc:299] Negotiation successful
0502 14:09:20.693831 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":1553,"threads_started":1}
I20240502 14:09:20.699685 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: Bootstrap starting.
I20240502 14:09:20.705221 8690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:20.706856 8690 log.cc:824] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:20.711328 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: No bootstrap required, opened a new log
I20240502 14:09:20.731148 8690 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } }
I20240502 14:09:20.731822 8690 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:20.732018 8690 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec49fb205a1a4dd48eb2a23a86b932b5, State: Initialized, Role: FOLLOWER
I20240502 14:09:20.732692 8690 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } }
I20240502 14:09:20.733196 8690 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:20.733430 8690 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:20.733695 8690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:20.738222 8690 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } }
I20240502 14:09:20.738889 8690 leader_election.cc:302] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec49fb205a1a4dd48eb2a23a86b932b5; no voters:
I20240502 14:09:20.740600 8690 leader_election.cc:288] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:20.741304 8695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:20.743742 8695 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [term 1 LEADER]: Becoming Leader. State: Replica: ec49fb205a1a4dd48eb2a23a86b932b5, State: Running, Role: LEADER
I20240502 14:09:20.744544 8695 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } }
I20240502 14:09:20.745494 8690 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:20.757122 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec49fb205a1a4dd48eb2a23a86b932b5. Latest consensus state: current_term: 1 leader_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } } }
I20240502 14:09:20.757182 8696 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec49fb205a1a4dd48eb2a23a86b932b5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41153 } } }
I20240502 14:09:20.758023 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:20.758023 8696 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:20.761771 8703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:20.772517 8703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:20.788894 8703 catalog_manager.cc:1347] Generated new cluster ID: 3cf9bc68e367442190e9f3419a8b6c27
I20240502 14:09:20.789144 8703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:20.819362 8703 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:20.820785 8703 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:20.831432 8703 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: Generated new TSK 0
I20240502 14:09:20.832264 8703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:09:20.856505 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--builtin_ntp_servers=127.27.174.84:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:21.146312 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:21.146852 8714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:21.147361 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:21.175721 8714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:21.175966 8714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:21.176716 8714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:21.209949 8714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:21.211261 8714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:21.212951 8714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:21.230224 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:21.232466 8714 server_base.cc:1034] running on GCE node
W20240502 14:09:21.233139 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:21.231546 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:22.261957 8714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:22.264781 8714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:22.266207 8714 hybrid_clock.cc:648] HybridClock initialized: now 1714658962266179 us; error 50 us; skew 500 ppm
I20240502 14:09:22.267030 8714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:22.273983 8714 webserver.cc:460] Webserver started at http://127.27.174.65:34167/ using document root <none> and password file <none>
I20240502 14:09:22.275009 8714 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:22.275209 8714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:22.275673 8714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:22.282660 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "ec364eb49196433da50bd0174985f567"
format_stamp: "Formatted at 2024-05-02 14:09:22 on dist-test-slave-dlb8"
server_key: "f6821f68300facf696865bc419cd71cd"
server_key_iv: "b67b7cc02ed7f1f4f1bd23bb5e3851da"
server_key_version: "encryptionkey@0"
I20240502 14:09:22.283994 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "ec364eb49196433da50bd0174985f567"
format_stamp: "Formatted at 2024-05-02 14:09:22 on dist-test-slave-dlb8"
server_key: "f6821f68300facf696865bc419cd71cd"
server_key_iv: "b67b7cc02ed7f1f4f1bd23bb5e3851da"
server_key_version: "encryptionkey@0"
I20240502 14:09:22.291294 8714 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240502 14:09:22.296821 8730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:22.297879 8714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20240502 14:09:22.298203 8714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "ec364eb49196433da50bd0174985f567"
format_stamp: "Formatted at 2024-05-02 14:09:22 on dist-test-slave-dlb8"
server_key: "f6821f68300facf696865bc419cd71cd"
server_key_iv: "b67b7cc02ed7f1f4f1bd23bb5e3851da"
server_key_version: "encryptionkey@0"
I20240502 14:09:22.298558 8714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:22.347033 8714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:22.348448 8714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:22.348879 8714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:22.351917 8714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:22.356225 8714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:22.356420 8714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:22.356657 8714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:22.356806 8714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:22.492954 8714 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:45107
I20240502 14:09:22.493062 8842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:45107 every 8 connection(s)
I20240502 14:09:22.495402 8714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:22.499272 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8714
I20240502 14:09:22.499749 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:22.501191 28345 external_mini_cluster.cc:1428] Setting key dca835421a2586dcbcac71ee33e75be7
I20240502 14:09:22.506142 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--builtin_ntp_servers=127.27.174.84:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:22.515174 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:22.499635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:60241 (local address 127.27.174.126:41153)
0502 14:09:22.500745 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:22.500761 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:22.502093 (+ 1332us) server_negotiation.cc:407] Connection header received
0502 14:09:22.506820 (+ 4727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:22.506863 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:22.507024 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:22.507171 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:22.508785 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:22.509598 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:22.510963 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:22.511541 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:22.512551 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:22.512616 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:22.512632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:22.512673 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:22.512737 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:22.512788 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:22.512898 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:22.513169 (+ 271us) server_negotiation.cc:299] Negotiation successful
0502 14:09:22.513360 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":514,"threads_started":1}
I20240502 14:09:22.515911 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:22.499393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41153 (local address 127.27.174.65:60241)
0502 14:09:22.501861 (+ 2468us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:22.501926 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:22.506474 (+ 4548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:22.507471 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:22.507495 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:22.507627 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:22.508534 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:22.508574 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:22.509840 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:22.509860 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:22.510738 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:22.510776 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:22.511133 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:22.512165 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:22.512179 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:22.512231 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:22.512332 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:22.512941 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:22.512963 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:22.512979 (+ 16us) client_negotiation.cc:769] Sending connection context
0502 14:09:22.513125 (+ 146us) client_negotiation.cc:240] Negotiation successful
0502 14:09:22.513399 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":1465,"threads_started":1}
I20240502 14:09:22.517807 8843 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41153
I20240502 14:09:22.518297 8843 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:22.519603 8843 heartbeater.cc:511] Master 127.27.174.126:41153 requested a full tablet report, sending...
I20240502 14:09:22.522907 8655 ts_manager.cc:194] Registered new tserver with Master: ec364eb49196433da50bd0174985f567 (127.27.174.65:45107)
I20240502 14:09:22.525998 8655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:60241
W20240502 14:09:22.808313 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:22.808776 8847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:22.809229 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:22.837882 8847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:22.838162 8847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:22.838925 8847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:22.870676 8847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:22.871748 8847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:22.873229 8847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:22.889714 8853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:23.530009 8843 heartbeater.cc:503] Master 127.27.174.126:41153 was elected leader, sending a full tablet report...
W20240502 14:09:22.890255 8854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:22.891423 8847 server_base.cc:1034] running on GCE node
W20240502 14:09:22.890681 8856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:23.906611 8847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:23.909770 8847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:23.911237 8847 hybrid_clock.cc:648] HybridClock initialized: now 1714658963911188 us; error 91 us; skew 500 ppm
I20240502 14:09:23.912189 8847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:23.919554 8847 webserver.cc:460] Webserver started at http://127.27.174.66:36443/ using document root <none> and password file <none>
I20240502 14:09:23.920737 8847 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:23.921001 8847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:23.921538 8847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:23.929178 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "1223f9b6ce23487e83c8742c8b6131fc"
format_stamp: "Formatted at 2024-05-02 14:09:23 on dist-test-slave-dlb8"
server_key: "f2ee43e59ae1af2e5ab389d1e15fdc19"
server_key_iv: "086152a04679c4dfc918032a338c5889"
server_key_version: "encryptionkey@0"
I20240502 14:09:23.930707 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "1223f9b6ce23487e83c8742c8b6131fc"
format_stamp: "Formatted at 2024-05-02 14:09:23 on dist-test-slave-dlb8"
server_key: "f2ee43e59ae1af2e5ab389d1e15fdc19"
server_key_iv: "086152a04679c4dfc918032a338c5889"
server_key_version: "encryptionkey@0"
I20240502 14:09:23.940419 8847 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20240502 14:09:23.948655 8863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:23.949594 8847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20240502 14:09:23.949867 8847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "1223f9b6ce23487e83c8742c8b6131fc"
format_stamp: "Formatted at 2024-05-02 14:09:23 on dist-test-slave-dlb8"
server_key: "f2ee43e59ae1af2e5ab389d1e15fdc19"
server_key_iv: "086152a04679c4dfc918032a338c5889"
server_key_version: "encryptionkey@0"
I20240502 14:09:23.950274 8847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:23.994131 8847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:23.995536 8847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:23.995950 8847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:23.998242 8847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:24.002007 8847 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:24.002244 8847 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 14:09:24.002467 8847 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:24.002627 8847 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:24.127763 8847 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:39105
I20240502 14:09:24.127871 8976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:39105 every 8 connection(s)
I20240502 14:09:24.130241 8847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:24.140810 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8847
I20240502 14:09:24.141216 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:24.142241 28345 external_mini_cluster.cc:1428] Setting key d8c469cfb0cb85047099a3fbcb75f633
I20240502 14:09:24.147486 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--builtin_ntp_servers=127.27.174.84:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:24.150147 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:24.134405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58783 (local address 127.27.174.126:41153)
0502 14:09:24.135601 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:24.135639 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:24.137160 (+ 1521us) server_negotiation.cc:407] Connection header received
0502 14:09:24.141697 (+ 4537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:24.141716 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:24.141918 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:24.142089 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:24.143743 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:24.144565 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:24.145960 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:24.146674 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:24.147557 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:24.147624 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:24.147640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:24.147678 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:24.147742 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:24.147784 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:24.147918 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:24.148212 (+ 294us) server_negotiation.cc:299] Negotiation successful
0502 14:09:24.148390 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":467,"threads_started":1}
I20240502 14:09:24.150890 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:24.134715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41153 (local address 127.27.174.66:58783)
0502 14:09:24.136934 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:24.136999 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:24.141358 (+ 4359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:24.142390 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:24.142429 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:24.142549 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:24.143491 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:24.143530 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:24.144788 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:24.144825 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:24.145725 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:24.145768 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:24.146158 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:24.147147 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:24.147162 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:24.147223 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:24.147328 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:24.147947 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:24.147979 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:24.147997 (+ 18us) client_negotiation.cc:769] Sending connection context
0502 14:09:24.148145 (+ 148us) client_negotiation.cc:240] Negotiation successful
0502 14:09:24.148434 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":1158,"threads_started":1}
I20240502 14:09:24.152727 8977 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41153
I20240502 14:09:24.153199 8977 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:24.154381 8977 heartbeater.cc:511] Master 127.27.174.126:41153 requested a full tablet report, sending...
I20240502 14:09:24.156351 8655 ts_manager.cc:194] Registered new tserver with Master: 1223f9b6ce23487e83c8742c8b6131fc (127.27.174.66:39105)
I20240502 14:09:24.157508 8655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:58783
W20240502 14:09:24.435659 8981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:24.436195 8981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:24.436764 8981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:24.464946 8981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:24.465195 8981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:24.466084 8981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:24.497546 8981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:24.498703 8981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:24.500138 8981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:24.515798 8987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:25.160725 8977 heartbeater.cc:503] Master 127.27.174.126:41153 was elected leader, sending a full tablet report...
W20240502 14:09:24.516288 8988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:24.517426 8981 server_base.cc:1034] running on GCE node
W20240502 14:09:24.516824 8990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:25.552335 8981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:25.554513 8981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:25.555836 8981 hybrid_clock.cc:648] HybridClock initialized: now 1714658965555816 us; error 41 us; skew 500 ppm
I20240502 14:09:25.556617 8981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:25.562649 8981 webserver.cc:460] Webserver started at http://127.27.174.67:42155/ using document root <none> and password file <none>
I20240502 14:09:25.563555 8981 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:25.563763 8981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:25.564188 8981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:25.569352 8981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af"
format_stamp: "Formatted at 2024-05-02 14:09:25 on dist-test-slave-dlb8"
server_key: "10ea419af4fd142dd90f5b4422726c51"
server_key_iv: "9a6fc17b1af00a014189a6b5d790dbee"
server_key_version: "encryptionkey@0"
I20240502 14:09:25.570487 8981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af"
format_stamp: "Formatted at 2024-05-02 14:09:25 on dist-test-slave-dlb8"
server_key: "10ea419af4fd142dd90f5b4422726c51"
server_key_iv: "9a6fc17b1af00a014189a6b5d790dbee"
server_key_version: "encryptionkey@0"
I20240502 14:09:25.577358 8981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 14:09:25.582736 8997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:25.583683 8981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:09:25.583961 8981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af"
format_stamp: "Formatted at 2024-05-02 14:09:25 on dist-test-slave-dlb8"
server_key: "10ea419af4fd142dd90f5b4422726c51"
server_key_iv: "9a6fc17b1af00a014189a6b5d790dbee"
server_key_version: "encryptionkey@0"
I20240502 14:09:25.584303 8981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:25.639573 8981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:25.640982 8981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:25.641415 8981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:25.644018 8981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:25.648139 8981 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:25.648326 8981 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:25.648527 8981 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:25.648653 8981 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:25.783479 8981 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:46787
I20240502 14:09:25.783596 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:46787 every 8 connection(s)
I20240502 14:09:25.785982 8981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:25.792470 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 8981
I20240502 14:09:25.792932 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:25.793908 28345 external_mini_cluster.cc:1428] Setting key 3ac06bb0ded73e07f325716e0858467b
I20240502 14:09:25.805757 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.789831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:59565 (local address 127.27.174.126:41153)
0502 14:09:25.791418 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:25.791455 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:25.792641 (+ 1186us) server_negotiation.cc:407] Connection header received
0502 14:09:25.797328 (+ 4687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:25.797358 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:25.797534 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:25.797676 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:25.799326 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.800248 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.801514 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.802115 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.803027 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:25.803101 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:25.803118 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:25.803159 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:25.803227 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:25.803265 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:25.803386 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:25.803688 (+ 302us) server_negotiation.cc:299] Negotiation successful
0502 14:09:25.803881 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":515,"threads_started":1}
I20240502 14:09:25.806419 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.790072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41153 (local address 127.27.174.67:59565)
0502 14:09:25.792408 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:25.792480 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:25.796988 (+ 4508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:25.797976 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:25.798000 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:25.798155 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:25.799079 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.799118 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.800472 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.800492 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:25.801286 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.801325 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.801666 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.802608 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:25.802623 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:25.802672 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:25.802789 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:25.803426 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:25.803453 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:25.803471 (+ 18us) client_negotiation.cc:769] Sending connection context
0502 14:09:25.803623 (+ 152us) client_negotiation.cc:240] Negotiation successful
0502 14:09:25.803912 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":1327,"threads_started":1}
I20240502 14:09:25.808035 9110 heartbeater.cc:347] Connected to a master server at 127.27.174.126:41153
I20240502 14:09:25.808367 9110 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:25.809377 9110 heartbeater.cc:511] Master 127.27.174.126:41153 requested a full tablet report, sending...
I20240502 14:09:25.811547 8654 ts_manager.cc:194] Registered new tserver with Master: fcb8c1e0f17f410ab97aa127e2e6b5af (127.27.174.67:46787)
I20240502 14:09:25.813071 8654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:59565
I20240502 14:09:25.825634 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:25.846217 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.837219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.27.174.126:41153)
0502 14:09:25.837814 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:25.837838 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:25.838676 (+ 838us) server_negotiation.cc:407] Connection header received
0502 14:09:25.838998 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:25.839015 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:25.839191 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:25.839334 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:25.840565 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.841380 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.842780 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.843345 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.844155 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:25.844226 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:25.844246 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:25.844289 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:25.844351 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:25.844400 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:25.844557 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:25.844810 (+ 253us) server_negotiation.cc:299] Negotiation successful
0502 14:09:25.845038 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20240502 14:09:25.851696 8654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:25.871496 8654 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:25.918367 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.897753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45107 (local address 127.0.0.1:49092)
0502 14:09:25.907256 (+ 9503us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:25.907315 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:25.907621 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:25.909647 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:25.909690 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:25.909824 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:25.911085 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.911183 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.913754 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.913788 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:25.915652 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.915692 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.916064 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.916217 (+ 153us) client_negotiation.cc:769] Sending connection context
0502 14:09:25.916389 (+ 172us) client_negotiation.cc:240] Negotiation successful
0502 14:09:25.916678 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8701,"thread_start_us":12652,"threads_started":1}
I20240502 14:09:25.919400 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.897798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.27.174.65:45107)
0502 14:09:25.901290 (+ 3492us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:25.901330 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:25.907901 (+ 6571us) server_negotiation.cc:407] Connection header received
0502 14:09:25.908173 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:25.908212 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:25.908679 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:25.908936 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:25.911510 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.913450 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.915962 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.917275 (+ 1313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.917602 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:25.917831 (+ 229us) server_negotiation.cc:299] Negotiation successful
0502 14:09:25.918074 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2643,"thread_start_us":2219,"threads_started":1}
I20240502 14:09:25.921651 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.899884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:46787 (local address 127.0.0.1:52496)
0502 14:09:25.911768 (+ 11884us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:25.911820 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:25.912069 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:25.913229 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:25.913258 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:25.913376 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:25.914295 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.914340 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.917436 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.917467 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:25.919258 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.919308 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.919617 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.919695 (+ 78us) client_negotiation.cc:769] Sending connection context
0502 14:09:25.919809 (+ 114us) client_negotiation.cc:240] Negotiation successful
0502 14:09:25.919998 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11256,"thread_start_us":6234,"threads_started":1}
I20240502 14:09:25.922896 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.901064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.27.174.67:46787)
0502 14:09:25.908052 (+ 6988us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:25.908082 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:25.911983 (+ 3901us) server_negotiation.cc:407] Connection header received
0502 14:09:25.912379 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:25.912414 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:25.912724 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:25.912907 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:25.914657 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.917143 (+ 2486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.919525 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.920645 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.920930 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:25.921150 (+ 220us) server_negotiation.cc:299] Negotiation successful
0502 14:09:25.921374 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6333,"thread_start_us":5790,"threads_started":1}
I20240502 14:09:25.924032 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.895713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:39105 (local address 127.0.0.1:45434)
0502 14:09:25.898789 (+ 3076us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:25.898889 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:25.899257 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:25.912850 (+ 13593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:25.912880 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:25.912976 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:25.913960 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.914019 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.917980 (+ 3961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.918011 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:25.919754 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:25.919807 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.920079 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.920157 (+ 78us) client_negotiation.cc:769] Sending connection context
0502 14:09:25.920271 (+ 114us) client_negotiation.cc:240] Negotiation successful
0502 14:09:25.920475 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":3105,"threads_started":1}
I20240502 14:09:25.926330 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:25.895948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.27.174.66:39105)
0502 14:09:25.912119 (+ 16171us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:25.912149 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:25.912209 (+ 60us) server_negotiation.cc:407] Connection header received
0502 14:09:25.912369 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:25.912386 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:25.912626 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:25.912790 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:25.915445 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.917738 (+ 2293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:25.922710 (+ 4972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:25.923839 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:25.924120 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:25.924357 (+ 237us) server_negotiation.cc:299] Negotiation successful
0502 14:09:25.924874 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15482,"thread_start_us":15145,"threads_started":1}
I20240502 14:09:25.926290 9045 tablet_service.cc:1467] Processing CreateTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e (DEFAULT_TABLE table=test-table [id=04f5a4ed03884802873ce66b7aaad467]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:25.926297 8778 tablet_service.cc:1467] Processing CreateTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e (DEFAULT_TABLE table=test-table [id=04f5a4ed03884802873ce66b7aaad467]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:25.928010 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:25.928840 9045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:25.930001 8911 tablet_service.cc:1467] Processing CreateTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e (DEFAULT_TABLE table=test-table [id=04f5a4ed03884802873ce66b7aaad467]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:25.931591 8911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:25.950497 9129 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Bootstrap starting.
I20240502 14:09:25.954792 9130 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Bootstrap starting.
I20240502 14:09:25.957362 9129 tablet_bootstrap.cc:654] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:25.959996 9129 log.cc:824] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:25.960397 9131 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Bootstrap starting.
I20240502 14:09:25.961719 9130 tablet_bootstrap.cc:654] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:25.964058 9130 log.cc:824] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:25.965490 9129 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: No bootstrap required, opened a new log
I20240502 14:09:25.966157 9129 ts_tablet_manager.cc:1367] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.003s
I20240502 14:09:25.967231 9131 tablet_bootstrap.cc:654] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:25.969182 9131 log.cc:824] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:25.970161 9130 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: No bootstrap required, opened a new log
I20240502 14:09:25.970681 9130 ts_tablet_manager.cc:1367] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20240502 14:09:25.977823 9131 tablet_bootstrap.cc:492] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: No bootstrap required, opened a new log
I20240502 14:09:25.978333 9131 ts_tablet_manager.cc:1367] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20240502 14:09:25.987789 9129 raft_consensus.cc:348] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:25.988454 9129 raft_consensus.cc:374] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:25.988709 9129 raft_consensus.cc:729] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb8c1e0f17f410ab97aa127e2e6b5af, State: Initialized, Role: FOLLOWER
I20240502 14:09:25.989562 9129 consensus_queue.cc:260] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:25.993172 9110 heartbeater.cc:503] Master 127.27.174.126:41153 was elected leader, sending a full tablet report...
I20240502 14:09:25.993801 9129 ts_tablet_manager.cc:1398] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Time spent starting tablet: real 0.027s user 0.022s sys 0.004s
I20240502 14:09:25.999204 9130 raft_consensus.cc:348] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.000124 9130 raft_consensus.cc:374] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:26.000408 9130 raft_consensus.cc:729] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec364eb49196433da50bd0174985f567, State: Initialized, Role: FOLLOWER
I20240502 14:09:26.001271 9130 consensus_queue.cc:260] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.005314 9130 ts_tablet_manager.cc:1398] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Time spent starting tablet: real 0.034s user 0.024s sys 0.009s
I20240502 14:09:26.005102 9131 raft_consensus.cc:348] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.006018 9131 raft_consensus.cc:374] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:26.006390 9131 raft_consensus.cc:729] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1223f9b6ce23487e83c8742c8b6131fc, State: Initialized, Role: FOLLOWER
I20240502 14:09:26.007407 9131 consensus_queue.cc:260] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.011178 9131 ts_tablet_manager.cc:1398] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Time spent starting tablet: real 0.033s user 0.025s sys 0.005s
W20240502 14:09:26.040441 9111 tablet.cc:2369] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:09:26.137188 8978 tablet.cc:2369] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:09:26.253391 8844 tablet.cc:2369] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:26.258500 9135 raft_consensus.cc:482] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:26.258966 9135 raft_consensus.cc:504] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.261400 9135 leader_election.cc:288] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1223f9b6ce23487e83c8742c8b6131fc (127.27.174.66:39105), ec364eb49196433da50bd0174985f567 (127.27.174.65:45107)
I20240502 14:09:26.269580 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.261734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:39105 (local address 127.27.174.67:53313)
0502 14:09:26.262756 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:26.262811 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:26.263073 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:26.263984 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:26.264005 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:26.264103 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:26.264645 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:26.264697 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.266532 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:26.266562 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:26.268075 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:26.268112 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.268343 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:26.268405 (+ 62us) client_negotiation.cc:769] Sending connection context
0502 14:09:26.268490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0502 14:09:26.268618 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"spinlock_wait_cycles":17664}
I20240502 14:09:26.271219 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.261914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:53313 (local address 127.27.174.66:39105)
0502 14:09:26.262676 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:26.262707 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:26.263310 (+ 603us) server_negotiation.cc:407] Connection header received
0502 14:09:26.263480 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:26.263509 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:26.263764 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:26.263922 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:26.264926 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.266318 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:26.268317 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.269447 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:26.269580 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:26.269754 (+ 174us) server_negotiation.cc:299] Negotiation successful
0502 14:09:26.269953 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20240502 14:09:26.273021 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.262467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56585 (local address 127.27.174.65:45107)
0502 14:09:26.263311 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:26.263338 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:26.264119 (+ 781us) server_negotiation.cc:407] Connection header received
0502 14:09:26.264482 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:26.264518 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:26.264746 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:26.264912 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:26.266401 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.267639 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:26.269887 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.271033 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:26.271183 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:26.271943 (+ 760us) server_negotiation.cc:299] Negotiation successful
0502 14:09:26.272157 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20240502 14:09:26.272873 8931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f8cd3b1f1d4ea4896dc1782d6cb28e" candidate_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1223f9b6ce23487e83c8742c8b6131fc" is_pre_election: true
I20240502 14:09:26.273615 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.261900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45107 (local address 127.27.174.67:56585)
0502 14:09:26.263932 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:26.263989 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:26.264238 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:26.264952 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:26.264994 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:26.265090 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:26.266142 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:26.266207 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.267873 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:26.267909 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:26.269583 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:26.269631 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.271455 (+ 1824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:26.271571 (+ 116us) client_negotiation.cc:769] Sending connection context
0502 14:09:26.271772 (+ 201us) client_negotiation.cc:240] Negotiation successful
0502 14:09:26.271910 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":1122,"threads_started":1}
I20240502 14:09:26.273821 8931 raft_consensus.cc:2454] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb8c1e0f17f410ab97aa127e2e6b5af in term 0.
I20240502 14:09:26.275089 9000 leader_election.cc:302] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1223f9b6ce23487e83c8742c8b6131fc, fcb8c1e0f17f410ab97aa127e2e6b5af; no voters:
I20240502 14:09:26.275238 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f8cd3b1f1d4ea4896dc1782d6cb28e" candidate_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec364eb49196433da50bd0174985f567" is_pre_election: true
I20240502 14:09:26.275965 9135 raft_consensus.cc:2781] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:26.276085 8798 raft_consensus.cc:2454] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb8c1e0f17f410ab97aa127e2e6b5af in term 0.
I20240502 14:09:26.276221 9135 raft_consensus.cc:482] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:26.276458 9135 raft_consensus.cc:3037] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:26.281240 9135 raft_consensus.cc:504] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.283126 9135 leader_election.cc:288] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [CANDIDATE]: Term 1 election: Requested vote from peers 1223f9b6ce23487e83c8742c8b6131fc (127.27.174.66:39105), ec364eb49196433da50bd0174985f567 (127.27.174.65:45107)
I20240502 14:09:26.283434 8931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f8cd3b1f1d4ea4896dc1782d6cb28e" candidate_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1223f9b6ce23487e83c8742c8b6131fc"
I20240502 14:09:26.283852 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f8cd3b1f1d4ea4896dc1782d6cb28e" candidate_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec364eb49196433da50bd0174985f567"
I20240502 14:09:26.283982 8931 raft_consensus.cc:3037] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:26.284401 8798 raft_consensus.cc:3037] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:26.291079 8931 raft_consensus.cc:2454] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb8c1e0f17f410ab97aa127e2e6b5af in term 1.
I20240502 14:09:26.291278 8798 raft_consensus.cc:2454] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb8c1e0f17f410ab97aa127e2e6b5af in term 1.
I20240502 14:09:26.292089 9000 leader_election.cc:302] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1223f9b6ce23487e83c8742c8b6131fc, fcb8c1e0f17f410ab97aa127e2e6b5af; no voters:
I20240502 14:09:26.292636 9135 raft_consensus.cc:2781] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:26.294140 9135 raft_consensus.cc:686] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 1 LEADER]: Becoming Leader. State: Replica: fcb8c1e0f17f410ab97aa127e2e6b5af, State: Running, Role: LEADER
I20240502 14:09:26.294868 9135 consensus_queue.cc:237] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } }
I20240502 14:09:26.305792 8655 catalog_manager.cc:5484] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af reported cstate change: term changed from 0 to 1, leader changed from <none> to fcb8c1e0f17f410ab97aa127e2e6b5af (127.27.174.67). New cstate: current_term: 1 leader_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcb8c1e0f17f410ab97aa127e2e6b5af" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 46787 } health_report { overall_health: HEALTHY } } }
I20240502 14:09:26.389851 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.382663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.27.174.67:46787)
0502 14:09:26.383445 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:26.383475 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:26.383826 (+ 351us) server_negotiation.cc:407] Connection header received
0502 14:09:26.384147 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:26.384164 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:26.384379 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:26.384522 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:26.385324 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.386151 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:26.387347 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:26.387663 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:26.387820 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:26.388313 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:26.388459 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:26.388720 (+ 261us) server_negotiation.cc:299] Negotiation successful
0502 14:09:26.388832 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20240502 14:09:26.401221 8798 raft_consensus.cc:1261] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P ec364eb49196433da50bd0174985f567 [term 1 FOLLOWER]: Refusing update from remote peer fcb8c1e0f17f410ab97aa127e2e6b5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:26.401429 8931 raft_consensus.cc:1261] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 1 FOLLOWER]: Refusing update from remote peer fcb8c1e0f17f410ab97aa127e2e6b5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:26.402629 9135 consensus_queue.cc:1035] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec364eb49196433da50bd0174985f567" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 45107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:26.403348 9139 consensus_queue.cc:1035] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1223f9b6ce23487e83c8742c8b6131fc" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:26.425149 9143 mvcc.cc:205] Tried to move back new op lower bound from 7023243126365253632 to 7023243125950521344. Current Snapshot: MvccSnapshot[applied={T|T < 7023243126365253632}]
I20240502 14:09:26.460455 8654 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:26.460922 8654 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:26.469259 8654 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ec49fb205a1a4dd48eb2a23a86b932b5: Sending DeleteTablet for 3 replicas of tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e
I20240502 14:09:26.471537 8911 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:26 UTC) from {username='slave'} at 127.0.0.1:45434
I20240502 14:09:26.471666 8778 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:26 UTC) from {username='slave'} at 127.0.0.1:49092
I20240502 14:09:26.473418 9045 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:26 UTC) from {username='slave'} at 127.0.0.1:52496
I20240502 14:09:26.474411 9153 tablet_replica.cc:330] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: stopping tablet replica
I20240502 14:09:26.475345 9153 raft_consensus.cc:2229] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:26.476334 9153 raft_consensus.cc:2258] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:26.477279 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8714
I20240502 14:09:26.480098 9153 ts_tablet_manager.cc:1875] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:26.490624 9155 tablet_replica.cc:330] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: stopping tablet replica
I20240502 14:09:26.491462 9155 raft_consensus.cc:2229] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:26.492522 9155 raft_consensus.cc:2258] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:26.496282 9155 ts_tablet_manager.cc:1875] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:26.497135 9153 ts_tablet_manager.cc:1888] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:26.497576 9153 log.cc:1196] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/e3f8cd3b1f1d4ea4896dc1782d6cb28e
I20240502 14:09:26.498593 9153 ts_tablet_manager.cc:1909] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P 1223f9b6ce23487e83c8742c8b6131fc: Deleting consensus metadata
I20240502 14:09:26.504063 8641 catalog_manager.cc:4830] TS 1223f9b6ce23487e83c8742c8b6131fc (127.27.174.66:39105): tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e (table test-table [id=04f5a4ed03884802873ce66b7aaad467]) successfully deleted
I20240502 14:09:26.508611 9155 ts_tablet_manager.cc:1888] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:26.508941 9155 log.cc:1196] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/e3f8cd3b1f1d4ea4896dc1782d6cb28e
I20240502 14:09:26.509549 9155 ts_tablet_manager.cc:1909] T e3f8cd3b1f1d4ea4896dc1782d6cb28e P fcb8c1e0f17f410ab97aa127e2e6b5af: Deleting consensus metadata
I20240502 14:09:26.512053 8641 catalog_manager.cc:4830] TS fcb8c1e0f17f410ab97aa127e2e6b5af (127.27.174.67:46787): tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e (table test-table [id=04f5a4ed03884802873ce66b7aaad467]) successfully deleted
W20240502 14:09:26.513932 8639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:45107 (error 108)
I20240502 14:09:26.515195 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8847
I20240502 14:09:26.528537 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:26.518711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:45107 (local address 127.0.0.1:49098)
0502 14:09:26.527416 (+ 8705us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:26.527709 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7909,"thread_start_us":7648,"threads_started":1}
W20240502 14:09:26.529177 8639 catalog_manager.cc:4573] TS ec364eb49196433da50bd0174985f567 (127.27.174.65:45107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3f8cd3b1f1d4ea4896dc1782d6cb28e: Network error: Client connection negotiation failed: client connection to 127.27.174.65:45107: connect: Connection refused (error 111)
I20240502 14:09:26.543766 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8981
I20240502 14:09:26.568441 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 8622
2024-05-02T14:09:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (7544 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:09:26 dist-test-slave-dlb8 krb5kdc[9160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:09:26 dist-test-slave-dlb8 krb5kdc[9160](info): set up 2 sockets
May 02 14:09:26 dist-test-slave-dlb8 krb5kdc[9160](info): commencing operation
krb5kdc: starting...
W20240502 14:09:27.556959 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:09:27 dist-test-slave-dlb8 krb5kdc[9160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:09:27.766005 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33357
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:28.051002 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:28.051589 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:28.052124 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:28.080883 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:28.081226 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:28.081542 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:28.081807 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:28.082048 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:28.115679 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:28.116833 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:28.117857 9174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:28.132849 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:28.133447 9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:28.134138 9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:28.134735 9174 server_base.cc:1034] running on GCE node
I20240502 14:09:29.194973 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:29.197562 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:29.198889 9174 hybrid_clock.cc:648] HybridClock initialized: now 1714658969198859 us; error 66 us; skew 500 ppm
I20240502 14:09:29.199635 9174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:29 dist-test-slave-dlb8 krb5kdc[9160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658969, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:29.210393 9174 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:09:29.215447 9174 webserver.cc:460] Webserver started at http://127.27.174.126:39729/ using document root <none> and password file <none>
I20240502 14:09:29.216349 9174 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:29.216590 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:29.217007 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:29.222096 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "f892566bc5f24a88a14f426c89def1c7"
format_stamp: "Formatted at 2024-05-02 14:09:29 on dist-test-slave-dlb8"
server_key: "4448de22297646478f9044027412eb4e"
server_key_iv: "c75a5cbabeef81157a7c10ee27caf372"
server_key_version: "encryptionkey@0"
I20240502 14:09:29.223287 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "f892566bc5f24a88a14f426c89def1c7"
format_stamp: "Formatted at 2024-05-02 14:09:29 on dist-test-slave-dlb8"
server_key: "4448de22297646478f9044027412eb4e"
server_key_iv: "c75a5cbabeef81157a7c10ee27caf372"
server_key_version: "encryptionkey@0"
I20240502 14:09:29.230190 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 14:09:29.235280 9191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:29.236300 9174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:09:29.236627 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "f892566bc5f24a88a14f426c89def1c7"
format_stamp: "Formatted at 2024-05-02 14:09:29 on dist-test-slave-dlb8"
server_key: "4448de22297646478f9044027412eb4e"
server_key_iv: "c75a5cbabeef81157a7c10ee27caf372"
server_key_version: "encryptionkey@0"
I20240502 14:09:29.236963 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:29.284644 9174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:29.289834 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:29.290298 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:29.355625 9174 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:33357
I20240502 14:09:29.355712 9242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:33357 every 8 connection(s)
I20240502 14:09:29.358249 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:29.361150 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9174
I20240502 14:09:29.361682 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:29.363224 28345 external_mini_cluster.cc:1428] Setting key 6e62f408035c6c6da5ba6e285e38c164
I20240502 14:09:29.364285 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:09:29 dist-test-slave-dlb8 krb5kdc[9160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:29.390926 9243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: Bootstrap starting.
I20240502 14:09:29.396584 9243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:29.398685 9243 log.cc:824] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:29.399490 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:29.366213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.27.174.126:33357)
0502 14:09:29.369068 (+ 2855us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:29.369105 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:29.369156 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:09:29.374273 (+ 5117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:29.374312 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:29.374653 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:29.374974 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:29.378762 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:29.380126 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:29.381981 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:29.382427 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:29.388708 (+ 6281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:29.388802 (+ 94us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:29.388830 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:29.388924 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:29.392931 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:29.394390 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:29.394412 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:29.394445 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:29.394625 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:29.395228 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:29.395246 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:29.395269 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:29.395847 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:29.396185 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:29.397028 (+ 843us) server_negotiation.cc:299] Negotiation successful
0502 14:09:29.397459 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2339,"thread_start_us":1799,"threads_started":1}
I20240502 14:09:29.404053 9243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: No bootstrap required, opened a new log
I20240502 14:09:29.423228 9243 raft_consensus.cc:348] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } }
I20240502 14:09:29.423930 9243 raft_consensus.cc:374] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:29.424175 9243 raft_consensus.cc:729] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f892566bc5f24a88a14f426c89def1c7, State: Initialized, Role: FOLLOWER
I20240502 14:09:29.424891 9243 consensus_queue.cc:260] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } }
I20240502 14:09:29.425413 9243 raft_consensus.cc:388] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:29.425678 9243 raft_consensus.cc:482] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:29.425971 9243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:29.431298 9243 raft_consensus.cc:504] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } }
I20240502 14:09:29.432005 9243 leader_election.cc:302] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f892566bc5f24a88a14f426c89def1c7; no voters:
I20240502 14:09:29.433710 9243 leader_election.cc:288] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:29.434123 9248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:29.437830 9248 raft_consensus.cc:686] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [term 1 LEADER]: Becoming Leader. State: Replica: f892566bc5f24a88a14f426c89def1c7, State: Running, Role: LEADER
I20240502 14:09:29.438891 9243 sys_catalog.cc:564] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:29.438745 9248 consensus_queue.cc:237] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } }
I20240502 14:09:29.449777 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f892566bc5f24a88a14f426c89def1c7. Latest consensus state: current_term: 1 leader_uuid: "f892566bc5f24a88a14f426c89def1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } } }
I20240502 14:09:29.450214 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f892566bc5f24a88a14f426c89def1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f892566bc5f24a88a14f426c89def1c7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 33357 } } }
I20240502 14:09:29.450881 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:29.450997 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:29.453800 9256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:29.465871 9256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:29.482697 9256 catalog_manager.cc:1347] Generated new cluster ID: dfc66c3aa67245e887a1a5aba385e6b9
I20240502 14:09:29.482969 9256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:29.498688 9256 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:29.500707 9256 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:29.516652 9256 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: Generated new TSK 0
I20240502 14:09:29.517738 9256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:09:29.579864 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:29.877202 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:29.877740 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:29.878374 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:29.908736 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:29.909085 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:29.909965 9269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:29.943948 9269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:29.945039 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:29.946081 9269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:29.964386 9276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:29.965991 9269 server_base.cc:1034] running on GCE node
W20240502 14:09:29.967813 9275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:29.966593 9278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:31.019379 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:31.022133 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:31.023567 9269 hybrid_clock.cc:648] HybridClock initialized: now 1714658971023535 us; error 72 us; skew 500 ppm
I20240502 14:09:31.024294 9269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:31 dist-test-slave-dlb8 krb5kdc[9160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:31.030006 9269 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:09:31.035729 9269 webserver.cc:460] Webserver started at http://127.27.174.65:37463/ using document root <none> and password file <none>
I20240502 14:09:31.036620 9269 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:31.036844 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:31.037240 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:31.042165 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "8c8217164d514218b42cf71ec3420576"
format_stamp: "Formatted at 2024-05-02 14:09:31 on dist-test-slave-dlb8"
server_key: "357ee8026497ebcbc17c429dd2d5c025"
server_key_iv: "8caee0b2640b6091cc4006297be29810"
server_key_version: "encryptionkey@0"
I20240502 14:09:31.043205 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "8c8217164d514218b42cf71ec3420576"
format_stamp: "Formatted at 2024-05-02 14:09:31 on dist-test-slave-dlb8"
server_key: "357ee8026497ebcbc17c429dd2d5c025"
server_key_iv: "8caee0b2640b6091cc4006297be29810"
server_key_version: "encryptionkey@0"
I20240502 14:09:31.050379 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240502 14:09:31.055934 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:31.057155 9269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:09:31.057466 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "8c8217164d514218b42cf71ec3420576"
format_stamp: "Formatted at 2024-05-02 14:09:31 on dist-test-slave-dlb8"
server_key: "357ee8026497ebcbc17c429dd2d5c025"
server_key_iv: "8caee0b2640b6091cc4006297be29810"
server_key_version: "encryptionkey@0"
I20240502 14:09:31.057824 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:31.115247 9269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:31.120852 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:31.121315 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:31.124056 9269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:31.128628 9269 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:31.128837 9269 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:31.129077 9269 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:31.129249 9269 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:31.261674 9269 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39959
I20240502 14:09:31.261775 9398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39959 every 8 connection(s)
I20240502 14:09:31.264295 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:31.274309 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9269
I20240502 14:09:31.274744 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:31.275745 28345 external_mini_cluster.cc:1428] Setting key 1f54c2284ebdc1e1eb5668b7f8ffea0f
May 02 14:09:31 dist-test-slave-dlb8 krb5kdc[9160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:31.296177 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:31.268423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33357 (local address 127.27.174.65:39371)
0502 14:09:31.271300 (+ 2877us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:31.271366 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:31.275625 (+ 4259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:31.276667 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:31.276695 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:31.277411 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:31.278606 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:31.278657 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:31.280266 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:31.280289 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:31.281086 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:31.281120 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:31.281389 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:31.282204 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:31.282265 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:31.284963 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:31.290791 (+ 5828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:31.290811 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:31.290833 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:31.291411 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:31.292019 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:31.292038 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:31.292051 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:31.292227 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:31.293119 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:31.293136 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:31.293507 (+ 371us) client_negotiation.cc:769] Sending connection context
0502 14:09:31.293781 (+ 274us) client_negotiation.cc:240] Negotiation successful
0502 14:09:31.294235 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2133,"thread_start_us":1588,"threads_started":1}
I20240502 14:09:31.297093 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:31.268102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:39371 (local address 127.27.174.126:33357)
0502 14:09:31.269193 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:31.269227 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:31.271492 (+ 2265us) server_negotiation.cc:407] Connection header received
0502 14:09:31.275975 (+ 4483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:31.276005 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:31.276258 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:31.276467 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:31.278853 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:31.280039 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:31.281310 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:31.281905 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:31.285226 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:31.285328 (+ 102us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:31.285362 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:31.285544 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:31.290595 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:31.291616 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:31.291657 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:31.291686 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:31.291863 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:31.292400 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:31.292445 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:31.292473 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:31.292952 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:31.293172 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:31.294073 (+ 901us) server_negotiation.cc:299] Negotiation successful
0502 14:09:31.294595 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":481,"threads_started":1}
I20240502 14:09:31.298753 9399 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33357
I20240502 14:09:31.299082 9399 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
I20240502 14:09:31.299968 9399 heartbeater.cc:511] Master 127.27.174.126:33357 requested a full tablet report, sending...
I20240502 14:09:31.302379 9208 ts_manager.cc:194] Registered new tserver with Master: 8c8217164d514218b42cf71ec3420576 (127.27.174.65:39959)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:09:31.304198 9208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:39371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:09:31.329841 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:31.612690 9405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:31.613162 9405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:31.613750 9405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:31.643843 9405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:31.644153 9405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:31.644989 9405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:31.679385 9405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:31.680465 9405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:31.681450 9405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:31.697440 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:32.307471 9399 heartbeater.cc:503] Master 127.27.174.126:33357 was elected leader, sending a full tablet report...
W20240502 14:09:31.697999 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:31.699487 9405 server_base.cc:1034] running on GCE node
W20240502 14:09:31.698751 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:32.709311 9405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:32.711517 9405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:32.712889 9405 hybrid_clock.cc:648] HybridClock initialized: now 1714658972712848 us; error 52 us; skew 500 ppm
I20240502 14:09:32.713711 9405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:32 dist-test-slave-dlb8 krb5kdc[9160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658972, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:32.719585 9405 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:09:32.725112 9405 webserver.cc:460] Webserver started at http://127.27.174.66:33549/ using document root <none> and password file <none>
I20240502 14:09:32.726006 9405 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:32.726267 9405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:32.726709 9405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:32.731580 9405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "7512727279a44b5890949164cc2354b1"
format_stamp: "Formatted at 2024-05-02 14:09:32 on dist-test-slave-dlb8"
server_key: "e78a6ddf3ba3e972009ba8b5cef614eb"
server_key_iv: "28f58e368bafa98ed9192502dd8c0e18"
server_key_version: "encryptionkey@0"
I20240502 14:09:32.732679 9405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "7512727279a44b5890949164cc2354b1"
format_stamp: "Formatted at 2024-05-02 14:09:32 on dist-test-slave-dlb8"
server_key: "e78a6ddf3ba3e972009ba8b5cef614eb"
server_key_iv: "28f58e368bafa98ed9192502dd8c0e18"
server_key_version: "encryptionkey@0"
I20240502 14:09:32.739643 9405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 14:09:32.745198 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:32.746240 9405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:09:32.746551 9405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "7512727279a44b5890949164cc2354b1"
format_stamp: "Formatted at 2024-05-02 14:09:32 on dist-test-slave-dlb8"
server_key: "e78a6ddf3ba3e972009ba8b5cef614eb"
server_key_iv: "28f58e368bafa98ed9192502dd8c0e18"
server_key_version: "encryptionkey@0"
I20240502 14:09:32.746941 9405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:32.808497 9405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:32.812976 9405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:32.813403 9405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:32.815639 9405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:32.819305 9405 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:32.819479 9405 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:32.819684 9405 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:32.819818 9405 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:32.954015 9405 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41037
I20240502 14:09:32.954124 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41037 every 8 connection(s)
I20240502 14:09:32.956408 9405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:32.961141 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9405
I20240502 14:09:32.961714 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:32.963397 28345 external_mini_cluster.cc:1428] Setting key cda047f51189c3582ab1829fe4dc3ec1
May 02 14:09:32 dist-test-slave-dlb8 krb5kdc[9160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658972, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:09:32.990339 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:32.961106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:46019 (local address 127.27.174.126:33357)
0502 14:09:32.962546 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:32.962560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:32.964168 (+ 1608us) server_negotiation.cc:407] Connection header received
0502 14:09:32.968855 (+ 4687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:32.968890 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:32.969131 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:32.969343 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:32.971750 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:32.973092 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:32.974423 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:32.975017 (+ 594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:32.979175 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:32.979249 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:32.979265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:32.979355 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:32.983313 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:32.984733 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:32.984774 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:32.984800 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:32.984988 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:32.985803 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:32.985844 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:32.985876 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:32.986498 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:32.986755 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:32.987809 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0502 14:09:32.988228 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":733,"threads_started":1}
I20240502 14:09:32.990970 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:32.961514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33357 (local address 127.27.174.66:46019)
0502 14:09:32.963953 (+ 2439us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:32.964013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:32.968548 (+ 4535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:32.969531 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:32.969559 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:32.970484 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:32.971484 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:32.971524 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:32.973331 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:32.973353 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:32.974172 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:32.974205 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:32.974599 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:32.975383 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:32.975447 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:32.978918 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:32.983555 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:32.983589 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:32.983620 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:32.984475 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:32.985208 (+ 733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:32.985256 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:32.985278 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:32.985542 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:32.986735 (+ 1193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:32.986799 (+ 64us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:32.987262 (+ 463us) client_negotiation.cc:769] Sending connection context
0502 14:09:32.987599 (+ 337us) client_negotiation.cc:240] Negotiation successful
0502 14:09:32.988173 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1899,"thread_start_us":1419,"threads_started":1}
I20240502 14:09:32.992698 9535 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33357
I20240502 14:09:32.993036 9535 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:32.993922 9535 heartbeater.cc:511] Master 127.27.174.126:33357 requested a full tablet report, sending...
I20240502 14:09:32.995940 9208 ts_manager.cc:194] Registered new tserver with Master: 7512727279a44b5890949164cc2354b1 (127.27.174.66:41037)
I20240502 14:09:32.997140 9208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:46019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:09:33.014349 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:33.293287 9541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:33.293710 9541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:33.294303 9541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:33.323330 9541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:33.323685 9541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:33.324605 9541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:33.358289 9541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:33.359372 9541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:33.360423 9541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:33.377146 9547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:33.999769 9535 heartbeater.cc:503] Master 127.27.174.126:33357 was elected leader, sending a full tablet report...
W20240502 14:09:33.377585 9548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:33.379062 9541 server_base.cc:1034] running on GCE node
W20240502 14:09:33.378499 9550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:34.407085 9541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:34.409298 9541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:34.410653 9541 hybrid_clock.cc:648] HybridClock initialized: now 1714658974410635 us; error 50 us; skew 500 ppm
I20240502 14:09:34.411417 9541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:34 dist-test-slave-dlb8 krb5kdc[9160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658974, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:34.423094 9541 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:09:34.428197 9541 webserver.cc:460] Webserver started at http://127.27.174.67:45297/ using document root <none> and password file <none>
I20240502 14:09:34.429052 9541 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:34.429255 9541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:34.429672 9541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:34.434521 9541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "478dd913de0f4e5fbd9834b2a53030a5"
format_stamp: "Formatted at 2024-05-02 14:09:34 on dist-test-slave-dlb8"
server_key: "b0b903612ce837b04c76319c01a70b32"
server_key_iv: "62d94a90cb332241eccca7e98f09852e"
server_key_version: "encryptionkey@0"
I20240502 14:09:34.435607 9541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "478dd913de0f4e5fbd9834b2a53030a5"
format_stamp: "Formatted at 2024-05-02 14:09:34 on dist-test-slave-dlb8"
server_key: "b0b903612ce837b04c76319c01a70b32"
server_key_iv: "62d94a90cb332241eccca7e98f09852e"
server_key_version: "encryptionkey@0"
I20240502 14:09:34.442517 9541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:09:34.448308 9558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:34.449292 9541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240502 14:09:34.449592 9541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "478dd913de0f4e5fbd9834b2a53030a5"
format_stamp: "Formatted at 2024-05-02 14:09:34 on dist-test-slave-dlb8"
server_key: "b0b903612ce837b04c76319c01a70b32"
server_key_iv: "62d94a90cb332241eccca7e98f09852e"
server_key_version: "encryptionkey@0"
I20240502 14:09:34.449924 9541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:34.518354 9541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:34.523733 9541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:34.524169 9541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:34.526618 9541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:34.530349 9541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:34.530546 9541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:34.530773 9541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:34.530928 9541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:34.660039 9541 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41657
I20240502 14:09:34.660138 9671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41657 every 8 connection(s)
I20240502 14:09:34.662562 9541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:34.665755 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9541
I20240502 14:09:34.666379 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:34.667923 28345 external_mini_cluster.cc:1428] Setting key 9a93294b06c21d9a665c1bb62b8d2118
May 02 14:09:34 dist-test-slave-dlb8 krb5kdc[9160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658974, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:34.696430 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.667035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:48155 (local address 127.27.174.126:33357)
0502 14:09:34.668441 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.668460 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.670140 (+ 1680us) server_negotiation.cc:407] Connection header received
0502 14:09:34.675397 (+ 5257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.675416 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.675595 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.675726 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:34.677890 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.678935 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.680410 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.681038 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.684936 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:34.685018 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:34.685043 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:34.685183 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:34.690269 (+ 5086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:34.691570 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:34.691603 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:34.691617 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:34.691741 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:34.692442 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:34.692460 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:34.692473 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:34.692828 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:34.693038 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.693906 (+ 868us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.694341 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":638,"threads_started":1}
I20240502 14:09:34.697034 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.667127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:33357 (local address 127.27.174.67:48155)
0502 14:09:34.669886 (+ 2759us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.669970 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.675108 (+ 5138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.675937 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.675960 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.676716 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:34.677622 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.677662 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.679182 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.679205 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.680179 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.680214 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.680556 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.681417 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:34.681478 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:34.684646 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:34.690484 (+ 5838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:34.690513 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:34.690533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:34.691380 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:34.691960 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:34.691986 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:34.692011 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:34.692273 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:34.693001 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:34.693030 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:34.693400 (+ 370us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.693700 (+ 300us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.694278 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2021,"thread_start_us":1485,"threads_started":1}
I20240502 14:09:34.698770 9672 heartbeater.cc:347] Connected to a master server at 127.27.174.126:33357
I20240502 14:09:34.699198 9672 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:34.700342 9672 heartbeater.cc:511] Master 127.27.174.126:33357 requested a full tablet report, sending...
I20240502 14:09:34.702270 9208 ts_manager.cc:194] Registered new tserver with Master: 478dd913de0f4e5fbd9834b2a53030a5 (127.27.174.67:41657)
I20240502 14:09:34.703442 9208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:48155
I20240502 14:09:34.714898 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:34.747344 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.729124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.27.174.126:33357)
0502 14:09:34.729646 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.729671 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.730557 (+ 886us) server_negotiation.cc:407] Connection header received
0502 14:09:34.730861 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.730880 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.731056 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.731196 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:34.733546 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.734378 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.735681 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.736241 (+ 560us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.738907 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:34.739026 (+ 119us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:34.739055 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:34.739146 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:34.742291 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:34.743333 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:34.743356 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:34.743370 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:34.743508 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:34.744081 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:34.744101 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:34.744114 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:34.744466 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:34.744822 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.745447 (+ 625us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.745828 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20240502 14:09:34.752943 9208 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:34.772560 9208 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:34.817418 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.797719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.27.174.65:39959)
0502 14:09:34.800096 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.800126 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.800912 (+ 786us) server_negotiation.cc:407] Connection header received
0502 14:09:34.801091 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.801121 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.801448 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.801640 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:34.807065 (+ 5425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.809753 (+ 2688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.812895 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.814232 (+ 1337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.814499 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.815228 (+ 729us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.815485 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":1263,"threads_started":1}
I20240502 14:09:34.818430 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.797547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39959 (local address 127.0.0.1:33474)
0502 14:09:34.800315 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.800393 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.800683 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.801888 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.801920 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.802017 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:34.806617 (+ 4600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.806706 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.810195 (+ 3489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.810237 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.812489 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.812561 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.814731 (+ 2170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.814842 (+ 111us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.816602 (+ 1760us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.816841 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2121,"thread_start_us":1833,"threads_started":1}
I20240502 14:09:34.821703 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.799163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41657 (local address 127.0.0.1:47258)
0502 14:09:34.810657 (+ 11494us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.810949 (+ 292us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.811236 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.813139 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.813181 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.813324 (+ 143us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:34.814503 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.814578 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.817551 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.817588 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.819382 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.819430 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.819743 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.819827 (+ 84us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.819952 (+ 125us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.820159 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9871,"thread_start_us":9602,"threads_started":1}
I20240502 14:09:34.823314 9334 tablet_service.cc:1467] Processing CreateTablet for tablet 53794f46552943c29aee0b29ce2fcd85 (DEFAULT_TABLE table=test-table [id=f9dc02f2a1b04eab81f556fd6c26b1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:34.824822 9334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53794f46552943c29aee0b29ce2fcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:34.824844 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.800825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.27.174.67:41657)
0502 14:09:34.803621 (+ 2796us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.803652 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.811563 (+ 7911us) server_negotiation.cc:407] Connection header received
0502 14:09:34.811808 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.811847 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.812277 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.812522 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:34.814896 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.817235 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.821287 (+ 4052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.822733 (+ 1446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.822977 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.823177 (+ 200us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.823390 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":3183,"threads_started":1}
I20240502 14:09:34.828994 9607 tablet_service.cc:1467] Processing CreateTablet for tablet 53794f46552943c29aee0b29ce2fcd85 (DEFAULT_TABLE table=test-table [id=f9dc02f2a1b04eab81f556fd6c26b1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:34.829798 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.802921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41037 (local address 127.0.0.1:42430)
0502 14:09:34.815797 (+ 12876us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.815878 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.816108 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.820949 (+ 4841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.820978 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.821076 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:34.822093 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.822142 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.824686 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.824716 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.827434 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.827469 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.827772 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.827852 (+ 80us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.827976 (+ 124us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.828168 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12247,"thread_start_us":13253,"threads_started":1}
I20240502 14:09:34.830755 9607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53794f46552943c29aee0b29ce2fcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:34.830838 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.803150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.27.174.66:41037)
0502 14:09:34.819640 (+ 16490us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.819667 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.819729 (+ 62us) server_negotiation.cc:407] Connection header received
0502 14:09:34.819910 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.819937 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.820280 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.820496 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:34.822422 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.824453 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.827695 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.828965 (+ 1270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.829170 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.829365 (+ 195us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.829590 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15839,"thread_start_us":17678,"threads_started":1}
I20240502 14:09:34.834484 9470 tablet_service.cc:1467] Processing CreateTablet for tablet 53794f46552943c29aee0b29ce2fcd85 (DEFAULT_TABLE table=test-table [id=f9dc02f2a1b04eab81f556fd6c26b1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:34.835711 9470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53794f46552943c29aee0b29ce2fcd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:34.855151 9691 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Bootstrap starting.
I20240502 14:09:34.857524 9692 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Bootstrap starting.
I20240502 14:09:34.858302 9693 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Bootstrap starting.
I20240502 14:09:34.862360 9692 tablet_bootstrap.cc:654] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:34.862365 9691 tablet_bootstrap.cc:654] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:34.864547 9692 log.cc:824] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:34.864552 9691 log.cc:824] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:34.864787 9693 tablet_bootstrap.cc:654] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:34.867141 9693 log.cc:824] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:34.870393 9692 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: No bootstrap required, opened a new log
I20240502 14:09:34.870762 9692 ts_tablet_manager.cc:1367] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240502 14:09:34.870970 9691 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: No bootstrap required, opened a new log
I20240502 14:09:34.871425 9691 ts_tablet_manager.cc:1367] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20240502 14:09:34.873124 9693 tablet_bootstrap.cc:492] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: No bootstrap required, opened a new log
I20240502 14:09:34.873540 9693 ts_tablet_manager.cc:1367] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240502 14:09:34.891283 9691 raft_consensus.cc:348] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.891950 9691 raft_consensus.cc:374] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:34.892197 9691 raft_consensus.cc:729] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8217164d514218b42cf71ec3420576, State: Initialized, Role: FOLLOWER
I20240502 14:09:34.892907 9691 consensus_queue.cc:260] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.899379 9691 ts_tablet_manager.cc:1398] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Time spent starting tablet: real 0.028s user 0.026s sys 0.001s
I20240502 14:09:34.899626 9692 raft_consensus.cc:348] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.900486 9692 raft_consensus.cc:374] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:34.900763 9692 raft_consensus.cc:729] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7512727279a44b5890949164cc2354b1, State: Initialized, Role: FOLLOWER
I20240502 14:09:34.901582 9692 consensus_queue.cc:260] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.902194 9693 raft_consensus.cc:348] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.903096 9693 raft_consensus.cc:374] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:34.903450 9693 raft_consensus.cc:729] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478dd913de0f4e5fbd9834b2a53030a5, State: Initialized, Role: FOLLOWER
I20240502 14:09:34.904455 9693 consensus_queue.cc:260] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.906636 9692 ts_tablet_manager.cc:1398] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Time spent starting tablet: real 0.036s user 0.034s sys 0.001s
I20240502 14:09:34.908589 9672 heartbeater.cc:503] Master 127.27.174.126:33357 was elected leader, sending a full tablet report...
I20240502 14:09:34.909368 9693 ts_tablet_manager.cc:1398] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Time spent starting tablet: real 0.036s user 0.034s sys 0.003s
W20240502 14:09:34.916915 9673 tablet.cc:2369] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:34.928079 9699 raft_consensus.cc:482] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:34.928489 9699 raft_consensus.cc:504] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.930685 9699 leader_election.cc:288] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c8217164d514218b42cf71ec3420576 (127.27.174.65:39959), 7512727279a44b5890949164cc2354b1 (127.27.174.66:41037)
I20240502 14:09:34.937470 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.931058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39959 (local address 127.27.174.67:34843)
0502 14:09:34.931709 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.931747 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.931994 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.932904 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.932929 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.933011 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:34.933320 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.933349 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.934536 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.934555 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.935951 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.936003 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.936294 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.936359 (+ 65us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.936454 (+ 95us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.936587 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20240502 14:09:34.939054 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.931143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:34843 (local address 127.27.174.65:39959)
0502 14:09:34.931609 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.931630 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.931874 (+ 244us) server_negotiation.cc:407] Connection header received
0502 14:09:34.932248 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.932275 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.932510 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.932658 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:34.933549 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.934347 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.936226 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.937352 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.937480 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.937649 (+ 169us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.937824 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20240502 14:09:34.940485 9354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53794f46552943c29aee0b29ce2fcd85" candidate_uuid: "478dd913de0f4e5fbd9834b2a53030a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8217164d514218b42cf71ec3420576" is_pre_election: true
I20240502 14:09:34.941359 9354 raft_consensus.cc:2454] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 478dd913de0f4e5fbd9834b2a53030a5 in term 0.
I20240502 14:09:34.941957 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.931794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41037 (local address 127.27.174.67:57771)
0502 14:09:34.934121 (+ 2327us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:34.934152 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:34.934313 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:34.934908 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:34.934937 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:34.935025 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:34.935963 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.936005 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.937713 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.937987 (+ 274us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:34.939652 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:34.939686 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.939948 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.940025 (+ 77us) client_negotiation.cc:769] Sending connection context
0502 14:09:34.940142 (+ 117us) client_negotiation.cc:240] Negotiation successful
0502 14:09:34.940307 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":1629,"threads_started":1}
I20240502 14:09:34.942258 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:34.932054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:57771 (local address 127.27.174.66:41037)
0502 14:09:34.932732 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:34.932760 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:34.934253 (+ 1493us) server_negotiation.cc:407] Connection header received
0502 14:09:34.934531 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:34.934560 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:34.934725 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:34.934830 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:34.936218 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.937436 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:34.939874 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:34.940652 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:34.940751 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:34.940911 (+ 160us) server_negotiation.cc:299] Negotiation successful
0502 14:09:34.941090 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20240502 14:09:34.942853 9559 leader_election.cc:302] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478dd913de0f4e5fbd9834b2a53030a5, 8c8217164d514218b42cf71ec3420576; no voters:
I20240502 14:09:34.943593 9699 raft_consensus.cc:2781] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:34.943928 9699 raft_consensus.cc:482] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:34.944216 9699 raft_consensus.cc:3037] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:34.944604 9490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53794f46552943c29aee0b29ce2fcd85" candidate_uuid: "478dd913de0f4e5fbd9834b2a53030a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512727279a44b5890949164cc2354b1" is_pre_election: true
I20240502 14:09:34.945197 9490 raft_consensus.cc:2454] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 478dd913de0f4e5fbd9834b2a53030a5 in term 0.
I20240502 14:09:34.948674 9699 raft_consensus.cc:504] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
I20240502 14:09:34.949862 9699 leader_election.cc:288] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [CANDIDATE]: Term 1 election: Requested vote from peers 8c8217164d514218b42cf71ec3420576 (127.27.174.65:39959), 7512727279a44b5890949164cc2354b1 (127.27.174.66:41037)
I20240502 14:09:34.950491 9354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53794f46552943c29aee0b29ce2fcd85" candidate_uuid: "478dd913de0f4e5fbd9834b2a53030a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8217164d514218b42cf71ec3420576"
I20240502 14:09:34.950587 9490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53794f46552943c29aee0b29ce2fcd85" candidate_uuid: "478dd913de0f4e5fbd9834b2a53030a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7512727279a44b5890949164cc2354b1"
I20240502 14:09:34.950865 9354 raft_consensus.cc:3037] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:34.950942 9490 raft_consensus.cc:3037] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:34.954929 9490 raft_consensus.cc:2454] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478dd913de0f4e5fbd9834b2a53030a5 in term 1.
I20240502 14:09:34.955636 9559 leader_election.cc:302] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478dd913de0f4e5fbd9834b2a53030a5, 7512727279a44b5890949164cc2354b1; no voters:
I20240502 14:09:34.956183 9699 raft_consensus.cc:2781] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:34.957263 9354 raft_consensus.cc:2454] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 478dd913de0f4e5fbd9834b2a53030a5 in term 1.
I20240502 14:09:34.957958 9699 raft_consensus.cc:686] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 1 LEADER]: Becoming Leader. State: Replica: 478dd913de0f4e5fbd9834b2a53030a5, State: Running, Role: LEADER
I20240502 14:09:34.959098 9699 consensus_queue.cc:237] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } }
W20240502 14:09:34.962828 9536 tablet.cc:2369] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:34.970355 9207 catalog_manager.cc:5484] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 478dd913de0f4e5fbd9834b2a53030a5 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "478dd913de0f4e5fbd9834b2a53030a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "478dd913de0f4e5fbd9834b2a53030a5" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 } health_report { overall_health: UNKNOWN } } }
W20240502 14:09:35.022468 9400 tablet.cc:2369] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:35.029119 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:35.022295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.27.174.67:41657)
0502 14:09:35.022940 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:35.022958 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:35.023035 (+ 77us) server_negotiation.cc:407] Connection header received
0502 14:09:35.023442 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:35.023461 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:35.023712 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:35.023841 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:35.024637 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:35.025370 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:35.026666 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:35.027004 (+ 338us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:35.027323 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:35.027773 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:35.027917 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:35.028122 (+ 205us) server_negotiation.cc:299] Negotiation successful
0502 14:09:35.028197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20240502 14:09:35.040004 9490 raft_consensus.cc:1261] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 1 FOLLOWER]: Refusing update from remote peer 478dd913de0f4e5fbd9834b2a53030a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:35.040208 9354 raft_consensus.cc:1261] T 53794f46552943c29aee0b29ce2fcd85 P 8c8217164d514218b42cf71ec3420576 [term 1 FOLLOWER]: Refusing update from remote peer 478dd913de0f4e5fbd9834b2a53030a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:35.041433 9701 consensus_queue.cc:1035] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7512727279a44b5890949164cc2354b1" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:35.041996 9699 consensus_queue.cc:1035] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c8217164d514218b42cf71ec3420576" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:35.089397 9207 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:35.090018 9207 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:35.099243 9207 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f892566bc5f24a88a14f426c89def1c7: Sending DeleteTablet for 3 replicas of tablet 53794f46552943c29aee0b29ce2fcd85
I20240502 14:09:35.101122 9334 tablet_service.cc:1514] Processing DeleteTablet for tablet 53794f46552943c29aee0b29ce2fcd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:33474
I20240502 14:09:35.101411 9607 tablet_service.cc:1514] Processing DeleteTablet for tablet 53794f46552943c29aee0b29ce2fcd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:47258
I20240502 14:09:35.102551 9470 tablet_service.cc:1514] Processing DeleteTablet for tablet 53794f46552943c29aee0b29ce2fcd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:35 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:42430
I20240502 14:09:35.108675 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9269
I20240502 14:09:35.112145 9716 tablet_replica.cc:330] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: stopping tablet replica
I20240502 14:09:35.112936 9715 tablet_replica.cc:330] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: stopping tablet replica
I20240502 14:09:35.112967 9716 raft_consensus.cc:2229] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:35.113655 9716 raft_consensus.cc:2258] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:35.113885 9715 raft_consensus.cc:2229] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:35.114928 9715 raft_consensus.cc:2258] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:35.117497 9716 ts_tablet_manager.cc:1875] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:35.117869 9715 ts_tablet_manager.cc:1875] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:35.131953 9716 ts_tablet_manager.cc:1888] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:35.131953 9715 ts_tablet_manager.cc:1888] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:35.132341 9715 log.cc:1196] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/53794f46552943c29aee0b29ce2fcd85
I20240502 14:09:35.132489 9716 log.cc:1196] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/53794f46552943c29aee0b29ce2fcd85
I20240502 14:09:35.133165 9715 ts_tablet_manager.cc:1909] T 53794f46552943c29aee0b29ce2fcd85 P 478dd913de0f4e5fbd9834b2a53030a5: Deleting consensus metadata
I20240502 14:09:35.133370 9716 ts_tablet_manager.cc:1909] T 53794f46552943c29aee0b29ce2fcd85 P 7512727279a44b5890949164cc2354b1: Deleting consensus metadata
I20240502 14:09:35.136576 9192 catalog_manager.cc:4830] TS 7512727279a44b5890949164cc2354b1 (127.27.174.66:41037): tablet 53794f46552943c29aee0b29ce2fcd85 (table test-table [id=f9dc02f2a1b04eab81f556fd6c26b1c6]) successfully deleted
I20240502 14:09:35.136760 9194 catalog_manager.cc:4830] TS 478dd913de0f4e5fbd9834b2a53030a5 (127.27.174.67:41657): tablet 53794f46552943c29aee0b29ce2fcd85 (table test-table [id=f9dc02f2a1b04eab81f556fd6c26b1c6]) successfully deleted
W20240502 14:09:35.148401 9192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:39959 (error 108)
I20240502 14:09:35.149531 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9405
I20240502 14:09:35.154505 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:35.153016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39959 (local address 127.0.0.1:33480)
0502 14:09:35.153732 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:35.153935 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20240502 14:09:35.155162 9192 catalog_manager.cc:4573] TS 8c8217164d514218b42cf71ec3420576 (127.27.174.65:39959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53794f46552943c29aee0b29ce2fcd85: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39959: connect: Connection refused (error 111)
I20240502 14:09:35.186678 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9541
I20240502 14:09:35.210932 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:35.209677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39959 (local address 127.0.0.1:33486)
0502 14:09:35.210343 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:35.210518 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20240502 14:09:35.212030 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9174
2024-05-02T14:09:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (8644 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-02T14:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:35Z Disabled control of system clock
I20240502 14:09:35.295548 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:39973
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:35.577358 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:35.577924 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:35.578425 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:35.606199 9721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:35.606535 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:35.606767 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:35.607014 9721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:35.607287 9721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:35.639801 9721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:39973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:35.640975 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:35.642609 9721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:35.656914 9728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:35.657433 9727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:35.658528 9730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:36.665681 9729 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:09:36.666167 9721 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:09:36.669636 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:36.672600 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:36.673998 9721 hybrid_clock.cc:648] HybridClock initialized: now 1714658976673956 us; error 62 us; skew 500 ppm
I20240502 14:09:36.674782 9721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:36.681021 9721 webserver.cc:460] Webserver started at http://127.27.174.126:36895/ using document root <none> and password file <none>
I20240502 14:09:36.682005 9721 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:36.682260 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:36.682703 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:36.689563 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "4661b8bef0ae40f7ac8571b636e85854"
format_stamp: "Formatted at 2024-05-02 14:09:36 on dist-test-slave-dlb8"
server_key: "1f6e80eed68a2243e3fc800d7f353d3a"
server_key_iv: "c96a18528c57f0ee8e9c95b8f077b621"
server_key_version: "encryptionkey@0"
I20240502 14:09:36.690639 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "4661b8bef0ae40f7ac8571b636e85854"
format_stamp: "Formatted at 2024-05-02 14:09:36 on dist-test-slave-dlb8"
server_key: "1f6e80eed68a2243e3fc800d7f353d3a"
server_key_iv: "c96a18528c57f0ee8e9c95b8f077b621"
server_key_version: "encryptionkey@0"
I20240502 14:09:36.697216 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20240502 14:09:36.702435 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:36.703406 9721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:09:36.703696 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "4661b8bef0ae40f7ac8571b636e85854"
format_stamp: "Formatted at 2024-05-02 14:09:36 on dist-test-slave-dlb8"
server_key: "1f6e80eed68a2243e3fc800d7f353d3a"
server_key_iv: "c96a18528c57f0ee8e9c95b8f077b621"
server_key_version: "encryptionkey@0"
I20240502 14:09:36.704010 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:36.754539 9721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:36.755962 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:36.756449 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:36.823450 9721 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:39973
I20240502 14:09:36.823529 9788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:39973 every 8 connection(s)
I20240502 14:09:36.826155 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:36.831051 9789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:36.835763 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9721
I20240502 14:09:36.836198 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:36.837301 28345 external_mini_cluster.cc:1428] Setting key 3544aac4fca00869c9d6aa27551f1710
I20240502 14:09:36.854009 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: Bootstrap starting.
I20240502 14:09:36.857208 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:36.839627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.27.174.126:39973)
0502 14:09:36.842543 (+ 2916us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:36.842570 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:36.842625 (+ 55us) server_negotiation.cc:407] Connection header received
0502 14:09:36.846492 (+ 3867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:36.846530 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:36.846892 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:36.847211 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:36.848865 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:36.850324 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:36.852335 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:36.852750 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:36.853738 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:36.853828 (+ 90us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:36.853858 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:36.854672 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:36.854769 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:36.854808 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:36.854933 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:36.855311 (+ 378us) server_negotiation.cc:299] Negotiation successful
0502 14:09:36.855630 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2234,"thread_start_us":1797,"threads_started":1}
I20240502 14:09:36.859788 9789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:36.861472 9789 log.cc:824] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:36.866092 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: No bootstrap required, opened a new log
I20240502 14:09:36.884883 9789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } }
I20240502 14:09:36.885537 9789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:36.885818 9789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4661b8bef0ae40f7ac8571b636e85854, State: Initialized, Role: FOLLOWER
I20240502 14:09:36.886695 9789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } }
I20240502 14:09:36.887202 9789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:36.887519 9789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:36.887861 9789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:36.891754 9789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } }
I20240502 14:09:36.892412 9789 leader_election.cc:302] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4661b8bef0ae40f7ac8571b636e85854; no voters:
I20240502 14:09:36.894073 9789 leader_election.cc:288] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:36.894840 9794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:36.897639 9794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [term 1 LEADER]: Becoming Leader. State: Replica: 4661b8bef0ae40f7ac8571b636e85854, State: Running, Role: LEADER
I20240502 14:09:36.898351 9794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } }
I20240502 14:09:36.898864 9789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:36.910171 9795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4661b8bef0ae40f7ac8571b636e85854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } } }
I20240502 14:09:36.910349 9796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4661b8bef0ae40f7ac8571b636e85854. Latest consensus state: current_term: 1 leader_uuid: "4661b8bef0ae40f7ac8571b636e85854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4661b8bef0ae40f7ac8571b636e85854" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 39973 } } }
I20240502 14:09:36.910878 9795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:36.910878 9796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:36.914099 9803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:36.926445 9803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:36.941617 9803 catalog_manager.cc:1347] Generated new cluster ID: da51cb2b3eab43bd904bf073cdd418fd
I20240502 14:09:36.941918 9803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:36.957321 9803 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:36.958741 9803 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:36.971719 9803 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: Generated new TSK 0
I20240502 14:09:36.972580 9803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:09:36.994266 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--builtin_ntp_servers=127.27.174.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:37.277706 9813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:37.278214 9813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:37.278720 9813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:37.306123 9813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:37.306425 9813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:37.307262 9813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:37.338394 9813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:37.339534 9813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:37.341110 9813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:37.358307 9820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:37.360306 9822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:37.361941 9813 server_base.cc:1034] running on GCE node
W20240502 14:09:37.360982 9819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:38.412704 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:38.415618 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:38.417055 9813 hybrid_clock.cc:648] HybridClock initialized: now 1714658978416983 us; error 73 us; skew 500 ppm
I20240502 14:09:38.418159 9813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:38.425644 9813 webserver.cc:460] Webserver started at http://127.27.174.65:43517/ using document root <none> and password file <none>
I20240502 14:09:38.426879 9813 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:38.427140 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:38.427728 9813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:38.436872 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "bbfade3062394807babbcd6fa0c2d750"
format_stamp: "Formatted at 2024-05-02 14:09:38 on dist-test-slave-dlb8"
server_key: "d0da927fcacf1e78c2e6caf81795d541"
server_key_iv: "0018b7cb0b3af6fbbd75b6691bfe15e7"
server_key_version: "encryptionkey@0"
I20240502 14:09:38.438432 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "bbfade3062394807babbcd6fa0c2d750"
format_stamp: "Formatted at 2024-05-02 14:09:38 on dist-test-slave-dlb8"
server_key: "d0da927fcacf1e78c2e6caf81795d541"
server_key_iv: "0018b7cb0b3af6fbbd75b6691bfe15e7"
server_key_version: "encryptionkey@0"
I20240502 14:09:38.445817 9813 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240502 14:09:38.451926 9830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:38.452934 9813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:09:38.453212 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "bbfade3062394807babbcd6fa0c2d750"
format_stamp: "Formatted at 2024-05-02 14:09:38 on dist-test-slave-dlb8"
server_key: "d0da927fcacf1e78c2e6caf81795d541"
server_key_iv: "0018b7cb0b3af6fbbd75b6691bfe15e7"
server_key_version: "encryptionkey@0"
I20240502 14:09:38.453560 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:38.512104 9813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:38.513466 9813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:38.513870 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:38.516351 9813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:38.520434 9813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:38.520617 9813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:38.520874 9813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:38.521016 9813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:38.647413 9813 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:39455
I20240502 14:09:38.647514 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:39455 every 8 connection(s)
I20240502 14:09:38.649909 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:38.655417 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9813
I20240502 14:09:38.656036 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:38.657202 28345 external_mini_cluster.cc:1428] Setting key faf0b855e0e53452e8cce0d23dbfff6b
I20240502 14:09:38.663476 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--builtin_ntp_servers=127.27.174.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:38.672158 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:38.653785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33041 (local address 127.27.174.126:39973)
0502 14:09:38.655487 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:38.655520 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:38.657346 (+ 1826us) server_negotiation.cc:407] Connection header received
0502 14:09:38.662927 (+ 5581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:38.662966 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:38.663211 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:38.663405 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:38.665062 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:38.665931 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:38.667685 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:38.668326 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:38.669166 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:38.669271 (+ 105us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:38.669312 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:38.669392 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:38.669504 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:38.669570 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:38.669754 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:38.670007 (+ 253us) server_negotiation.cc:299] Negotiation successful
0502 14:09:38.670260 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":627,"threads_started":1}
I20240502 14:09:38.672389 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:38.654030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:39973 (local address 127.27.174.65:33041)
0502 14:09:38.657100 (+ 3070us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:38.657169 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:38.662583 (+ 5414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:38.663588 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:38.663612 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:38.663731 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:38.664755 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:38.664815 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:38.666188 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:38.666209 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:38.667389 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:38.667458 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:38.667842 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:38.668751 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:38.668766 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:38.668821 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:38.668920 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:38.669756 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:38.669776 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:38.669795 (+ 19us) client_negotiation.cc:769] Sending connection context
0502 14:09:38.669939 (+ 144us) client_negotiation.cc:240] Negotiation successful
0502 14:09:38.670212 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2203,"thread_start_us":1764,"threads_started":1}
I20240502 14:09:38.674345 9943 heartbeater.cc:347] Connected to a master server at 127.27.174.126:39973
I20240502 14:09:38.674711 9943 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:38.675680 9943 heartbeater.cc:511] Master 127.27.174.126:39973 requested a full tablet report, sending...
I20240502 14:09:38.678170 9754 ts_manager.cc:194] Registered new tserver with Master: bbfade3062394807babbcd6fa0c2d750 (127.27.174.65:39455)
I20240502 14:09:38.680003 9754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:33041
W20240502 14:09:38.957984 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:38.958498 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:38.958963 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:38.987097 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:38.987380 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:38.988133 9947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:39.019068 9947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:39.020192 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:39.021643 9947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:39.036183 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:39.683081 9943 heartbeater.cc:503] Master 127.27.174.126:39973 was elected leader, sending a full tablet report...
W20240502 14:09:39.036530 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:39.038532 9947 server_base.cc:1034] running on GCE node
W20240502 14:09:39.037833 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:40.046315 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:40.048523 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:40.049860 9947 hybrid_clock.cc:648] HybridClock initialized: now 1714658980049841 us; error 41 us; skew 500 ppm
I20240502 14:09:40.050570 9947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:40.061887 9947 webserver.cc:460] Webserver started at http://127.27.174.66:34857/ using document root <none> and password file <none>
I20240502 14:09:40.062793 9947 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:40.063011 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:40.063462 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:40.068415 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "6435000b405042fc83316437313e7b59"
format_stamp: "Formatted at 2024-05-02 14:09:40 on dist-test-slave-dlb8"
server_key: "d80a0f48b2306fd54311919acf29b12e"
server_key_iv: "fb9f53374d57b6de2c59869a86feda04"
server_key_version: "encryptionkey@0"
I20240502 14:09:40.069437 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "6435000b405042fc83316437313e7b59"
format_stamp: "Formatted at 2024-05-02 14:09:40 on dist-test-slave-dlb8"
server_key: "d80a0f48b2306fd54311919acf29b12e"
server_key_iv: "fb9f53374d57b6de2c59869a86feda04"
server_key_version: "encryptionkey@0"
I20240502 14:09:40.075970 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20240502 14:09:40.081218 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:40.082177 9947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:09:40.082466 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "6435000b405042fc83316437313e7b59"
format_stamp: "Formatted at 2024-05-02 14:09:40 on dist-test-slave-dlb8"
server_key: "d80a0f48b2306fd54311919acf29b12e"
server_key_iv: "fb9f53374d57b6de2c59869a86feda04"
server_key_version: "encryptionkey@0"
I20240502 14:09:40.082811 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:40.132433 9947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:40.133919 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:40.134370 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:40.136837 9947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:40.140741 9947 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:40.140947 9947 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:40.141185 9947 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:40.141340 9947 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:40.264027 9947 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:41391
I20240502 14:09:40.264120 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:41391 every 8 connection(s)
I20240502 14:09:40.266337 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:40.267158 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 9947
I20240502 14:09:40.267596 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:40.268579 28345 external_mini_cluster.cc:1428] Setting key f2202562981a45ff693bbbb0e5039b04
I20240502 14:09:40.275141 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--builtin_ntp_servers=127.27.174.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:40.288739 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:40.271479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:39973 (local address 127.27.174.66:58795)
0502 14:09:40.274286 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:40.274371 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:40.279066 (+ 4695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:40.280089 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:40.280125 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:40.280294 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:40.281582 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:40.281645 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:40.283602 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:40.283621 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:40.284400 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:40.284439 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:40.284697 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:40.285593 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:40.285609 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:40.285670 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:40.285773 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:40.286628 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:40.286659 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:40.286678 (+ 19us) client_negotiation.cc:769] Sending connection context
0502 14:09:40.286835 (+ 157us) client_negotiation.cc:240] Negotiation successful
0502 14:09:40.287061 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2007,"thread_start_us":1624,"threads_started":1}
I20240502 14:09:40.289052 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:40.271239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:58795 (local address 127.27.174.126:39973)
0502 14:09:40.275063 (+ 3824us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:40.275093 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:40.275146 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:09:40.279413 (+ 4267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:40.279443 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:40.279692 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:40.279876 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:40.281936 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:40.283373 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:40.284633 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:40.285371 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:40.285992 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:40.286133 (+ 141us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:40.286168 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:40.286260 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:40.286353 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:40.286430 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:40.286597 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:40.286903 (+ 306us) server_negotiation.cc:299] Negotiation successful
0502 14:09:40.287118 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3251,"thread_start_us":2971,"threads_started":1}
I20240502 14:09:40.290776 10076 heartbeater.cc:347] Connected to a master server at 127.27.174.126:39973
I20240502 14:09:40.291122 10076 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:40.292243 10076 heartbeater.cc:511] Master 127.27.174.126:39973 requested a full tablet report, sending...
I20240502 14:09:40.294281 9754 ts_manager.cc:194] Registered new tserver with Master: 6435000b405042fc83316437313e7b59 (127.27.174.66:41391)
I20240502 14:09:40.295392 9754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:58795
W20240502 14:09:40.559046 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:40.559554 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:40.560092 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:40.589368 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:40.589648 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:40.590476 10080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:40.623390 10080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:39973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:40.624569 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:40.626175 10080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:40.642745 10086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:41.298122 10076 heartbeater.cc:503] Master 127.27.174.126:39973 was elected leader, sending a full tablet report...
W20240502 14:09:40.643297 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:40.644596 10080 server_base.cc:1034] running on GCE node
W20240502 14:09:40.643936 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:41.674166 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:41.676221 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:41.677572 10080 hybrid_clock.cc:648] HybridClock initialized: now 1714658981677519 us; error 59 us; skew 500 ppm
I20240502 14:09:41.678501 10080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:41.684207 10080 webserver.cc:460] Webserver started at http://127.27.174.67:43671/ using document root <none> and password file <none>
I20240502 14:09:41.685074 10080 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:41.685266 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:41.685639 10080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:41.690626 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "756770ccf05f440ea029816d25dde040"
format_stamp: "Formatted at 2024-05-02 14:09:41 on dist-test-slave-dlb8"
server_key: "6ab2ad8421bb4ce9fc9da9a4e14d405a"
server_key_iv: "55a1ca5469d9227038e927cce9e7b912"
server_key_version: "encryptionkey@0"
I20240502 14:09:41.691906 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "756770ccf05f440ea029816d25dde040"
format_stamp: "Formatted at 2024-05-02 14:09:41 on dist-test-slave-dlb8"
server_key: "6ab2ad8421bb4ce9fc9da9a4e14d405a"
server_key_iv: "55a1ca5469d9227038e927cce9e7b912"
server_key_version: "encryptionkey@0"
I20240502 14:09:41.698850 10080 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 14:09:41.703995 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:41.704938 10080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:09:41.705228 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "756770ccf05f440ea029816d25dde040"
format_stamp: "Formatted at 2024-05-02 14:09:41 on dist-test-slave-dlb8"
server_key: "6ab2ad8421bb4ce9fc9da9a4e14d405a"
server_key_iv: "55a1ca5469d9227038e927cce9e7b912"
server_key_version: "encryptionkey@0"
I20240502 14:09:41.705601 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:41.759553 10080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:41.760979 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:41.761435 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:41.763881 10080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:41.767807 10080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:41.768021 10080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:41.768240 10080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:41.768388 10080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:41.896999 10080 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:41339
I20240502 14:09:41.897095 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:41339 every 8 connection(s)
I20240502 14:09:41.899516 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:41.908026 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10080
I20240502 14:09:41.908427 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:41.909407 28345 external_mini_cluster.cc:1428] Setting key 409887ae0b9166c3d6b7838ecb676a70
I20240502 14:09:41.919013 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.903267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:38191 (local address 127.27.174.126:39973)
0502 14:09:41.904278 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:41.904294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:41.906518 (+ 2224us) server_negotiation.cc:407] Connection header received
0502 14:09:41.911177 (+ 4659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:41.911201 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:41.911418 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:41.911600 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:41.913366 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.914594 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:41.915881 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.916211 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:41.917119 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:41.917196 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:41.917213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:41.917253 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:41.917333 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:41.917369 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:41.917476 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:41.917709 (+ 233us) server_negotiation.cc:299] Negotiation successful
0502 14:09:41.917859 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20240502 14:09:41.919535 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.903611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:39973 (local address 127.27.174.67:38191)
0502 14:09:41.906328 (+ 2717us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:41.906394 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:41.910872 (+ 4478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:41.911854 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:41.911884 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:41.912033 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:41.913092 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:41.913133 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.914801 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:41.914835 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:41.915689 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:41.915741 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.915988 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:41.916744 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:41.916760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:41.916812 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:41.916917 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:41.917505 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:41.917526 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:41.917545 (+ 19us) client_negotiation.cc:769] Sending connection context
0502 14:09:41.917675 (+ 130us) client_negotiation.cc:240] Negotiation successful
0502 14:09:41.917889 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1912,"thread_start_us":1516,"threads_started":1}
I20240502 14:09:41.920948 10209 heartbeater.cc:347] Connected to a master server at 127.27.174.126:39973
I20240502 14:09:41.921272 10209 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:41.922216 10209 heartbeater.cc:511] Master 127.27.174.126:39973 requested a full tablet report, sending...
I20240502 14:09:41.924049 9754 ts_manager.cc:194] Registered new tserver with Master: 756770ccf05f440ea029816d25dde040 (127.27.174.67:41339)
I20240502 14:09:41.925271 9754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:38191
I20240502 14:09:41.928062 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:41.951043 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.942188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.27.174.126:39973)
0502 14:09:41.942703 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:41.942730 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:41.943769 (+ 1039us) server_negotiation.cc:407] Connection header received
0502 14:09:41.944135 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:41.944153 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:41.944302 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:41.944437 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:41.945677 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.946489 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:41.947954 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:41.948495 (+ 541us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:41.949284 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:41.949366 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:41.949395 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:41.949440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:41.949498 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:41.949538 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:41.949699 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:41.949983 (+ 284us) server_negotiation.cc:299] Negotiation successful
0502 14:09:41.950115 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20240502 14:09:41.955984 9754 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:41.974279 9754 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:42.013757 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.997673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41391 (local address 127.0.0.1:39186)
0502 14:09:42.000016 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.000109 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:42.000427 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:42.006513 (+ 6086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:42.006573 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:42.006722 (+ 149us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:42.007679 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.007727 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.010348 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.010374 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:42.011538 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.011577 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.011848 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.011949 (+ 101us) client_negotiation.cc:769] Sending connection context
0502 14:09:42.012098 (+ 149us) client_negotiation.cc:240] Negotiation successful
0502 14:09:42.012338 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":1262,"threads_started":1}
I20240502 14:09:42.015655 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.999224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39455 (local address 127.0.0.1:52462)
0502 14:09:42.004070 (+ 4846us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.004302 (+ 232us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:42.004513 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:42.007270 (+ 2757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:42.007300 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:42.007416 (+ 116us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:42.008364 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.008409 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.010837 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.010870 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:42.012536 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.012592 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.012897 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.012975 (+ 78us) client_negotiation.cc:769] Sending connection context
0502 14:09:42.013087 (+ 112us) client_negotiation.cc:240] Negotiation successful
0502 14:09:42.013273 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4158,"mutex_wait_us":37,"thread_start_us":4863,"threads_started":1}
I20240502 14:09:42.018013 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.002889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:41339 (local address 127.0.0.1:39188)
0502 14:09:42.007925 (+ 5036us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.007999 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:42.008343 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:42.009603 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:42.009626 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:42.009712 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:42.010493 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.010530 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.013014 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.013045 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:42.015656 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.015707 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.016648 (+ 941us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.016707 (+ 59us) client_negotiation.cc:769] Sending connection context
0502 14:09:42.016801 (+ 94us) client_negotiation.cc:240] Negotiation successful
0502 14:09:42.016938 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4354,"thread_start_us":6172,"threads_started":1}
I20240502 14:09:42.018237 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.999713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.27.174.65:39455)
0502 14:09:42.006330 (+ 6617us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.006356 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.006403 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:09:42.006601 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.006634 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.006971 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.008621 (+ 1650us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:42.009034 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.010616 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.014337 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.016040 (+ 1703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.016323 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.016541 (+ 218us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.016793 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6056,"thread_start_us":5762,"threads_started":1}
I20240502 14:09:42.018509 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:41.998117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.27.174.66:41391)
0502 14:09:42.001331 (+ 3214us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.001360 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.001412 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:09:42.001596 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.001622 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.006154 (+ 4532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.006374 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:42.008026 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.010110 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.011777 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.012865 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.013151 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.013352 (+ 201us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.013577 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2628,"thread_start_us":2401,"threads_started":1}
I20240502 14:09:42.019357 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.003137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.27.174.67:41339)
0502 14:09:42.006057 (+ 2920us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.006087 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.008110 (+ 2023us) server_negotiation.cc:407] Connection header received
0502 14:09:42.008638 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.008663 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.008967 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.009133 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:42.010737 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.012770 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.015945 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.017142 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.017438 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.017667 (+ 229us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.017900 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2318,"thread_start_us":1842,"threads_started":1}
I20240502 14:09:42.022572 10011 tablet_service.cc:1467] Processing CreateTablet for tablet 38cc9dab251646ba945f0abdd48002c0 (DEFAULT_TABLE table=test-table [id=7a29791b551e40079fc0166ef63bde7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:42.022604 9878 tablet_service.cc:1467] Processing CreateTablet for tablet 38cc9dab251646ba945f0abdd48002c0 (DEFAULT_TABLE table=test-table [id=7a29791b551e40079fc0166ef63bde7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:42.023151 10144 tablet_service.cc:1467] Processing CreateTablet for tablet 38cc9dab251646ba945f0abdd48002c0 (DEFAULT_TABLE table=test-table [id=7a29791b551e40079fc0166ef63bde7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:42.024431 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cc9dab251646ba945f0abdd48002c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:42.024431 9878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cc9dab251646ba945f0abdd48002c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:42.024551 10144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cc9dab251646ba945f0abdd48002c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:42.045258 10229 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Bootstrap starting.
I20240502 14:09:42.050307 10230 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Bootstrap starting.
I20240502 14:09:42.052235 10231 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Bootstrap starting.
I20240502 14:09:42.053339 10229 tablet_bootstrap.cc:654] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:42.055409 10229 log.cc:824] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:42.056408 10231 tablet_bootstrap.cc:654] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:42.056639 10230 tablet_bootstrap.cc:654] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:42.058652 10231 log.cc:824] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:42.059265 10230 log.cc:824] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:42.061220 10229 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: No bootstrap required, opened a new log
I20240502 14:09:42.061769 10229 ts_tablet_manager.cc:1367] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20240502 14:09:42.063858 10231 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: No bootstrap required, opened a new log
I20240502 14:09:42.064278 10231 ts_tablet_manager.cc:1367] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 14:09:42.064811 10230 tablet_bootstrap.cc:492] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: No bootstrap required, opened a new log
I20240502 14:09:42.065228 10230 ts_tablet_manager.cc:1367] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 14:09:42.080744 10229 raft_consensus.cc:348] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.081327 10229 raft_consensus.cc:374] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:42.081591 10229 raft_consensus.cc:729] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756770ccf05f440ea029816d25dde040, State: Initialized, Role: FOLLOWER
I20240502 14:09:42.082360 10229 consensus_queue.cc:260] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.085320 10209 heartbeater.cc:503] Master 127.27.174.126:39973 was elected leader, sending a full tablet report...
I20240502 14:09:42.086001 10229 ts_tablet_manager.cc:1398] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Time spent starting tablet: real 0.024s user 0.016s sys 0.007s
I20240502 14:09:42.092217 10231 raft_consensus.cc:348] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.092820 10231 raft_consensus.cc:374] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:42.093036 10231 raft_consensus.cc:729] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbfade3062394807babbcd6fa0c2d750, State: Initialized, Role: FOLLOWER
I20240502 14:09:42.092801 10230 raft_consensus.cc:348] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.093621 10230 raft_consensus.cc:374] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:42.093883 10230 raft_consensus.cc:729] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6435000b405042fc83316437313e7b59, State: Initialized, Role: FOLLOWER
I20240502 14:09:42.093684 10231 consensus_queue.cc:260] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.094707 10230 consensus_queue.cc:260] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.098850 10230 ts_tablet_manager.cc:1398] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Time spent starting tablet: real 0.033s user 0.035s sys 0.000s
I20240502 14:09:42.099282 10231 ts_tablet_manager.cc:1398] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Time spent starting tablet: real 0.035s user 0.030s sys 0.005s
W20240502 14:09:42.153952 10210 tablet.cc:2369] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:09:42.157879 9944 tablet.cc:2369] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:42.196096 10235 raft_consensus.cc:482] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:42.196595 10235 raft_consensus.cc:504] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.199334 10235 leader_election.cc:288] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6435000b405042fc83316437313e7b59 (127.27.174.66:41391), bbfade3062394807babbcd6fa0c2d750 (127.27.174.65:39455)
I20240502 14:09:42.207929 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.199774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:41391 (local address 127.27.174.67:56511)
0502 14:09:42.200818 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.200864 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:42.201122 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:42.201911 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:42.201947 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:42.202029 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:42.202518 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.202581 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.203941 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.203973 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:42.205138 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.205174 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.205470 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.205555 (+ 85us) client_negotiation.cc:769] Sending connection context
0502 14:09:42.205671 (+ 116us) client_negotiation.cc:240] Negotiation successful
0502 14:09:42.205842 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20240502 14:09:42.208974 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.199866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:56511 (local address 127.27.174.66:41391)
0502 14:09:42.200548 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.200574 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.201000 (+ 426us) server_negotiation.cc:407] Connection header received
0502 14:09:42.201369 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.201397 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.201650 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.201825 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:42.202794 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.203725 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.206181 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.207362 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.207508 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.207685 (+ 177us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.207899 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20240502 14:09:42.209295 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.199874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39455 (local address 127.27.174.67:54005)
0502 14:09:42.202186 (+ 2312us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.202224 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:42.202396 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:42.203100 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:42.203117 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:42.203183 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:42.203882 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.203942 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.205433 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.205452 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:42.206832 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:42.206870 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.207180 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.207260 (+ 80us) client_negotiation.cc:769] Sending connection context
0502 14:09:42.207388 (+ 128us) client_negotiation.cc:240] Negotiation successful
0502 14:09:42.207582 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1659,"spinlock_wait_cycles":17664,"thread_start_us":1590,"threads_started":1}
I20240502 14:09:42.209996 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.200456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:54005 (local address 127.27.174.65:39455)
0502 14:09:42.201533 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.201550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.202551 (+ 1001us) server_negotiation.cc:407] Connection header received
0502 14:09:42.202669 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.202692 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.202927 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.203074 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:42.204165 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.205142 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.207081 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.208356 (+ 1275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.208478 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.208641 (+ 163us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.208818 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588}
I20240502 14:09:42.210491 10031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cc9dab251646ba945f0abdd48002c0" candidate_uuid: "756770ccf05f440ea029816d25dde040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6435000b405042fc83316437313e7b59" is_pre_election: true
I20240502 14:09:42.211272 10031 raft_consensus.cc:2454] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 756770ccf05f440ea029816d25dde040 in term 0.
I20240502 14:09:42.211326 9898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cc9dab251646ba945f0abdd48002c0" candidate_uuid: "756770ccf05f440ea029816d25dde040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfade3062394807babbcd6fa0c2d750" is_pre_election: true
I20240502 14:09:42.212014 9898 raft_consensus.cc:2454] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 756770ccf05f440ea029816d25dde040 in term 0.
I20240502 14:09:42.212256 10097 leader_election.cc:302] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6435000b405042fc83316437313e7b59, 756770ccf05f440ea029816d25dde040; no voters:
I20240502 14:09:42.212961 10235 raft_consensus.cc:2781] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:42.213269 10235 raft_consensus.cc:482] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:42.213517 10235 raft_consensus.cc:3037] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:42.218135 10235 raft_consensus.cc:504] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.219437 10235 leader_election.cc:288] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [CANDIDATE]: Term 1 election: Requested vote from peers 6435000b405042fc83316437313e7b59 (127.27.174.66:41391), bbfade3062394807babbcd6fa0c2d750 (127.27.174.65:39455)
I20240502 14:09:42.220000 10031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cc9dab251646ba945f0abdd48002c0" candidate_uuid: "756770ccf05f440ea029816d25dde040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6435000b405042fc83316437313e7b59"
I20240502 14:09:42.220222 9898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cc9dab251646ba945f0abdd48002c0" candidate_uuid: "756770ccf05f440ea029816d25dde040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfade3062394807babbcd6fa0c2d750"
I20240502 14:09:42.220376 10031 raft_consensus.cc:3037] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:42.220636 9898 raft_consensus.cc:3037] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:42.224570 10031 raft_consensus.cc:2454] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756770ccf05f440ea029816d25dde040 in term 1.
I20240502 14:09:42.224864 9898 raft_consensus.cc:2454] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756770ccf05f440ea029816d25dde040 in term 1.
I20240502 14:09:42.225349 10097 leader_election.cc:302] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6435000b405042fc83316437313e7b59, 756770ccf05f440ea029816d25dde040; no voters:
I20240502 14:09:42.225883 10235 raft_consensus.cc:2781] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:42.227123 10235 raft_consensus.cc:686] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 1 LEADER]: Becoming Leader. State: Replica: 756770ccf05f440ea029816d25dde040, State: Running, Role: LEADER
I20240502 14:09:42.227850 10235 consensus_queue.cc:237] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } }
I20240502 14:09:42.238674 9754 catalog_manager.cc:5484] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 756770ccf05f440ea029816d25dde040 (127.27.174.67). New cstate: current_term: 1 leader_uuid: "756770ccf05f440ea029816d25dde040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "756770ccf05f440ea029816d25dde040" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 41339 } health_report { overall_health: HEALTHY } } }
I20240502 14:09:42.270215 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.262771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.27.174.67:41339)
0502 14:09:42.263416 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:42.263440 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:42.263479 (+ 39us) server_negotiation.cc:407] Connection header received
0502 14:09:42.263714 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:42.263730 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:42.263933 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:42.264083 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:42.265050 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.265834 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:42.267276 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:42.267814 (+ 538us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:42.268201 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:42.268687 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:42.268869 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:42.269168 (+ 299us) server_negotiation.cc:299] Negotiation successful
0502 14:09:42.269267 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20240502 14:09:42.272925 10077 tablet.cc:2369] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:42.283403 10031 raft_consensus.cc:1261] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 1 FOLLOWER]: Refusing update from remote peer 756770ccf05f440ea029816d25dde040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:42.283651 9898 raft_consensus.cc:1261] T 38cc9dab251646ba945f0abdd48002c0 P bbfade3062394807babbcd6fa0c2d750 [term 1 FOLLOWER]: Refusing update from remote peer 756770ccf05f440ea029816d25dde040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:42.284793 10239 consensus_queue.cc:1035] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6435000b405042fc83316437313e7b59" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:09:42.285486 10235 consensus_queue.cc:1035] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbfade3062394807babbcd6fa0c2d750" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:42.322985 10244 mvcc.cc:205] Tried to move back new op lower bound from 7023243191417880576 to 7023243191211896832. Current Snapshot: MvccSnapshot[applied={T|T < 7023243191417880576}]
I20240502 14:09:42.333912 9753 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37690:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:42.334512 9753 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37690:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:42.344027 9753 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4661b8bef0ae40f7ac8571b636e85854: Sending DeleteTablet for 3 replicas of tablet 38cc9dab251646ba945f0abdd48002c0
I20240502 14:09:42.345953 10011 tablet_service.cc:1514] Processing DeleteTablet for tablet 38cc9dab251646ba945f0abdd48002c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:42 UTC) from {username='slave'} at 127.0.0.1:39186
I20240502 14:09:42.346252 9878 tablet_service.cc:1514] Processing DeleteTablet for tablet 38cc9dab251646ba945f0abdd48002c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:42 UTC) from {username='slave'} at 127.0.0.1:52462
I20240502 14:09:42.348114 10144 tablet_service.cc:1514] Processing DeleteTablet for tablet 38cc9dab251646ba945f0abdd48002c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:42 UTC) from {username='slave'} at 127.0.0.1:39188
I20240502 14:09:42.350328 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9813
I20240502 14:09:42.354632 10253 tablet_replica.cc:330] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: stopping tablet replica
I20240502 14:09:42.355118 10251 tablet_replica.cc:330] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: stopping tablet replica
I20240502 14:09:42.355533 10253 raft_consensus.cc:2229] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:42.355916 10251 raft_consensus.cc:2229] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:42.356626 10251 raft_consensus.cc:2258] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:42.356691 10253 raft_consensus.cc:2258] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:42.359777 10251 ts_tablet_manager.cc:1875] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:42.360114 10253 ts_tablet_manager.cc:1875] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:42.374262 10251 ts_tablet_manager.cc:1888] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:42.374771 10251 log.cc:1196] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/38cc9dab251646ba945f0abdd48002c0
I20240502 14:09:42.375733 10251 ts_tablet_manager.cc:1909] T 38cc9dab251646ba945f0abdd48002c0 P 6435000b405042fc83316437313e7b59: Deleting consensus metadata
I20240502 14:09:42.377108 10253 ts_tablet_manager.cc:1888] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:42.377601 10253 log.cc:1196] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/38cc9dab251646ba945f0abdd48002c0
I20240502 14:09:42.378515 10253 ts_tablet_manager.cc:1909] T 38cc9dab251646ba945f0abdd48002c0 P 756770ccf05f440ea029816d25dde040: Deleting consensus metadata
I20240502 14:09:42.379268 9738 catalog_manager.cc:4830] TS 6435000b405042fc83316437313e7b59 (127.27.174.66:41391): tablet 38cc9dab251646ba945f0abdd48002c0 (table test-table [id=7a29791b551e40079fc0166ef63bde7f]) successfully deleted
I20240502 14:09:42.381414 9738 catalog_manager.cc:4830] TS 756770ccf05f440ea029816d25dde040 (127.27.174.67:41339): tablet 38cc9dab251646ba945f0abdd48002c0 (table test-table [id=7a29791b551e40079fc0166ef63bde7f]) successfully deleted
I20240502 14:09:42.385869 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9947
W20240502 14:09:42.386034 9741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:39455 (error 108)
I20240502 14:09:42.391305 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:42.389576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:39455 (local address 127.0.0.1:52472)
0502 14:09:42.390451 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:42.390735 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
W20240502 14:09:42.391822 9741 catalog_manager.cc:4573] TS bbfade3062394807babbcd6fa0c2d750 (127.27.174.65:39455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38cc9dab251646ba945f0abdd48002c0: Network error: Client connection negotiation failed: client connection to 127.27.174.65:39455: connect: Connection refused (error 111)
I20240502 14:09:42.412822 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10080
I20240502 14:09:42.437525 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 9721
2024-05-02T14:09:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (7225 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:09:42 dist-test-slave-dlb8 krb5kdc[10257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:09:42 dist-test-slave-dlb8 krb5kdc[10257](info): set up 2 sockets
May 02 14:09:42 dist-test-slave-dlb8 krb5kdc[10257](info): commencing operation
krb5kdc: starting...
W20240502 14:09:43.413064 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:09:43 dist-test-slave-dlb8 krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:09:43.626299 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43251
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:43.914549 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:43.915148 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:43.915735 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:43.944705 10270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:43.945025 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:43.945349 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:43.945603 10270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:43.945828 10270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:43.978744 10270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:43251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:43251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:43.979802 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:43.980715 10270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:43.996124 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:43.996591 10279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:43.997241 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:43.997967 10270 server_base.cc:1034] running on GCE node
I20240502 14:09:45.029911 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:45.032984 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:45.034394 10270 hybrid_clock.cc:648] HybridClock initialized: now 1714658985034369 us; error 48 us; skew 500 ppm
I20240502 14:09:45.035101 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:45 dist-test-slave-dlb8 krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:45.040656 10270 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:09:45.045913 10270 webserver.cc:460] Webserver started at http://127.27.174.126:39585/ using document root <none> and password file <none>
I20240502 14:09:45.046747 10270 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:45.046958 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:45.047359 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:45.052326 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "373972bd046a4115a200069910bad2af"
format_stamp: "Formatted at 2024-05-02 14:09:45 on dist-test-slave-dlb8"
server_key: "508075e8879c63b0f0b3c8d68ca54578"
server_key_iv: "64e4a1d60185d6f8481c23a12c7a8815"
server_key_version: "encryptionkey@0"
I20240502 14:09:45.053326 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "373972bd046a4115a200069910bad2af"
format_stamp: "Formatted at 2024-05-02 14:09:45 on dist-test-slave-dlb8"
server_key: "508075e8879c63b0f0b3c8d68ca54578"
server_key_iv: "64e4a1d60185d6f8481c23a12c7a8815"
server_key_version: "encryptionkey@0"
I20240502 14:09:45.060014 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20240502 14:09:45.064980 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:45.065848 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:09:45.066164 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "373972bd046a4115a200069910bad2af"
format_stamp: "Formatted at 2024-05-02 14:09:45 on dist-test-slave-dlb8"
server_key: "508075e8879c63b0f0b3c8d68ca54578"
server_key_iv: "64e4a1d60185d6f8481c23a12c7a8815"
server_key_version: "encryptionkey@0"
I20240502 14:09:45.066494 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:45.121508 10270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:45.127236 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:45.127713 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:45.191507 10270 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:43251
I20240502 14:09:45.191571 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:43251 every 8 connection(s)
I20240502 14:09:45.194072 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:45.199270 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:45.201606 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10270
I20240502 14:09:45.202020 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:45.202987 28345 external_mini_cluster.cc:1428] Setting key 7aaa5fc2adb6499ada99e2fca68f6f52
I20240502 14:09:45.218649 10339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: Bootstrap starting.
May 02 14:09:45 dist-test-slave-dlb8 krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:45.224975 10339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:45.226508 10339 log.cc:824] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:45.231850 10339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: No bootstrap required, opened a new log
I20240502 14:09:45.235301 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:45.205195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.27.174.126:43251)
0502 14:09:45.207851 (+ 2656us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:45.207879 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:45.207930 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:09:45.211543 (+ 3613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:45.211578 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:45.211871 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:45.212167 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:45.215379 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:45.216627 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:45.218486 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:45.218863 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:45.224796 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:45.224891 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:45.224916 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:45.225012 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:45.229686 (+ 4674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:45.230927 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:45.230948 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:45.230980 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:45.231134 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:45.231694 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:45.231712 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:45.231724 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:45.232276 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:45.232572 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:45.233193 (+ 621us) server_negotiation.cc:299] Negotiation successful
0502 14:09:45.233569 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2133,"thread_start_us":1685,"threads_started":1}
I20240502 14:09:45.251534 10339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } }
I20240502 14:09:45.252207 10339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:45.252451 10339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373972bd046a4115a200069910bad2af, State: Initialized, Role: FOLLOWER
I20240502 14:09:45.253095 10339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } }
I20240502 14:09:45.253572 10339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:45.253824 10339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:45.254128 10339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:45.258414 10339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } }
I20240502 14:09:45.259084 10339 leader_election.cc:302] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 373972bd046a4115a200069910bad2af; no voters:
I20240502 14:09:45.260795 10339 leader_election.cc:288] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:45.261211 10344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:45.264127 10344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [term 1 LEADER]: Becoming Leader. State: Replica: 373972bd046a4115a200069910bad2af, State: Running, Role: LEADER
I20240502 14:09:45.264732 10344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } }
I20240502 14:09:45.265677 10339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:45.276345 10346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 373972bd046a4115a200069910bad2af. Latest consensus state: current_term: 1 leader_uuid: "373972bd046a4115a200069910bad2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } } }
I20240502 14:09:45.277428 10346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:45.279847 10345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "373972bd046a4115a200069910bad2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373972bd046a4115a200069910bad2af" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 43251 } } }
I20240502 14:09:45.280589 10345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:45.283671 10352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:45.294080 10352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:45.310492 10352 catalog_manager.cc:1347] Generated new cluster ID: f2d94fa8bbbc4bde884cd1ae18e3aa0c
I20240502 14:09:45.310767 10352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:45.324901 10352 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:45.326256 10352 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:45.337668 10352 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: Generated new TSK 0
I20240502 14:09:45.338618 10352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
Entry for principal kudu/127.27.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab.
I20240502 14:09:45.402336 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:45.697620 10365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:45.698153 10365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:45.698830 10365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:45.727577 10365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:45.727936 10365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:45.728842 10365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:45.762293 10365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.65.keytab
--principal=kudu/127.27.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:45.763456 10365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:45.764496 10365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:45.781899 10372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:45.784646 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:45.785173 10365 server_base.cc:1034] running on GCE node
W20240502 14:09:45.782559 10371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:46.842279 10365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:46.846632 10365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:46.848033 10365 hybrid_clock.cc:648] HybridClock initialized: now 1714658986847984 us; error 68 us; skew 500 ppm
I20240502 14:09:46.848836 10365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:46 dist-test-slave-dlb8 krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:46.854350 10365 init.cc:377] Logged in from keytab as kudu/127.27.174.65@KRBTEST.COM (short username kudu)
I20240502 14:09:46.860025 10365 webserver.cc:460] Webserver started at http://127.27.174.65:36435/ using document root <none> and password file <none>
I20240502 14:09:46.860886 10365 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:46.861074 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:46.861532 10365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:46.866598 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "24cb9f7b5a6f49c1be59c57400f8229b"
format_stamp: "Formatted at 2024-05-02 14:09:46 on dist-test-slave-dlb8"
server_key: "9190da943e3c05b721ba37eeae309ff2"
server_key_iv: "b6151ada2c78b931917694f1cb35009c"
server_key_version: "encryptionkey@0"
I20240502 14:09:46.867758 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "24cb9f7b5a6f49c1be59c57400f8229b"
format_stamp: "Formatted at 2024-05-02 14:09:46 on dist-test-slave-dlb8"
server_key: "9190da943e3c05b721ba37eeae309ff2"
server_key_iv: "b6151ada2c78b931917694f1cb35009c"
server_key_version: "encryptionkey@0"
I20240502 14:09:46.875033 10365 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240502 14:09:46.880710 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:46.882004 10365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:09:46.882355 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "24cb9f7b5a6f49c1be59c57400f8229b"
format_stamp: "Formatted at 2024-05-02 14:09:46 on dist-test-slave-dlb8"
server_key: "9190da943e3c05b721ba37eeae309ff2"
server_key_iv: "b6151ada2c78b931917694f1cb35009c"
server_key_version: "encryptionkey@0"
I20240502 14:09:46.882715 10365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:46.928695 10365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:46.934015 10365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:46.934527 10365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:46.936949 10365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:46.940712 10365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:46.940914 10365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:46.941149 10365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:46.941315 10365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:47.063117 10365 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40085
I20240502 14:09:47.063235 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40085 every 8 connection(s)
I20240502 14:09:47.065438 10365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:47.066478 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10365
I20240502 14:09:47.067112 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:47.068712 28345 external_mini_cluster.cc:1428] Setting key bbbaf0be14162f9d0b901dc4841ab5d8
May 02 14:09:47 dist-test-slave-dlb8 krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.65@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.66@KRBTEST.COM" created.
I20240502 14:09:47.100391 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:47.070252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:33629 (local address 127.27.174.126:43251)
0502 14:09:47.071867 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:47.071890 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:47.073269 (+ 1379us) server_negotiation.cc:407] Connection header received
0502 14:09:47.080012 (+ 6743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:47.080029 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:47.080176 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:47.080294 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:47.082856 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:47.084252 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:47.085610 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:47.086202 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:47.089787 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:47.089915 (+ 128us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:47.089950 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:47.090145 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:47.094238 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:47.095272 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:47.095313 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:47.095342 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:47.095509 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:47.096168 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:47.096190 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:47.096214 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:47.096593 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:47.096812 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:47.097798 (+ 986us) server_negotiation.cc:299] Negotiation successful
0502 14:09:47.098297 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":588,"threads_started":1}
I20240502 14:09:47.100899 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:47.070480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43251 (local address 127.27.174.65:33629)
0502 14:09:47.073039 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:47.073119 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:47.079667 (+ 6548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:47.080572 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:47.080610 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:47.081616 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:47.082594 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:47.082634 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:47.084539 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:47.084569 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:47.085381 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:47.085415 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:47.085679 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:47.086476 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:47.086538 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:47.089510 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:47.094453 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:47.094475 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:47.094507 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:47.095068 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:47.095675 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:47.095707 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:47.095731 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:47.096006 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:47.096794 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:47.096823 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:47.097218 (+ 395us) client_negotiation.cc:769] Sending connection context
0502 14:09:47.097533 (+ 315us) client_negotiation.cc:240] Negotiation successful
0502 14:09:47.098137 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"thread_start_us":1454,"threads_started":1}
I20240502 14:09:47.102841 10495 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43251
I20240502 14:09:47.103201 10495 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:47.104107 10495 heartbeater.cc:511] Master 127.27.174.126:43251 requested a full tablet report, sending...
I20240502 14:09:47.106431 10304 ts_manager.cc:194] Registered new tserver with Master: 24cb9f7b5a6f49c1be59c57400f8229b (127.27.174.65:40085)
I20240502 14:09:47.108187 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.65@KRBTEST.COM'} at 127.27.174.65:33629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
Entry for principal kudu/127.27.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab.
I20240502 14:09:47.121687 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:47.397351 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:47.397939 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:47.398728 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:47.428695 10501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:47.429119 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:47.430025 10501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:47.463094 10501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.66.keytab
--principal=kudu/127.27.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:47.464128 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:47.465081 10501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:47.480403 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:48.111269 10495 heartbeater.cc:503] Master 127.27.174.126:43251 was elected leader, sending a full tablet report...
W20240502 14:09:47.481016 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:47.482580 10501 server_base.cc:1034] running on GCE node
W20240502 14:09:47.481703 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:48.494539 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:48.496595 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:48.498051 10501 hybrid_clock.cc:648] HybridClock initialized: now 1714658988497994 us; error 48 us; skew 500 ppm
I20240502 14:09:48.499187 10501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:48 dist-test-slave-dlb8 krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658988, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:48.504873 10501 init.cc:377] Logged in from keytab as kudu/127.27.174.66@KRBTEST.COM (short username kudu)
I20240502 14:09:48.509656 10501 webserver.cc:460] Webserver started at http://127.27.174.66:39445/ using document root <none> and password file <none>
I20240502 14:09:48.510488 10501 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:48.510666 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:48.511090 10501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:48.515820 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "fd2250521136450bbb8c05fd8fbd6a36"
format_stamp: "Formatted at 2024-05-02 14:09:48 on dist-test-slave-dlb8"
server_key: "187453171098447b4ec0ffb6fa0301a5"
server_key_iv: "ed8c3efb768225909edc16e52c368da5"
server_key_version: "encryptionkey@0"
I20240502 14:09:48.516822 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "fd2250521136450bbb8c05fd8fbd6a36"
format_stamp: "Formatted at 2024-05-02 14:09:48 on dist-test-slave-dlb8"
server_key: "187453171098447b4ec0ffb6fa0301a5"
server_key_iv: "ed8c3efb768225909edc16e52c368da5"
server_key_version: "encryptionkey@0"
I20240502 14:09:48.523773 10501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20240502 14:09:48.529067 10519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:48.529924 10501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:09:48.530225 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "fd2250521136450bbb8c05fd8fbd6a36"
format_stamp: "Formatted at 2024-05-02 14:09:48 on dist-test-slave-dlb8"
server_key: "187453171098447b4ec0ffb6fa0301a5"
server_key_iv: "ed8c3efb768225909edc16e52c368da5"
server_key_version: "encryptionkey@0"
I20240502 14:09:48.530550 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:48.587358 10501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:48.592388 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:48.592801 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:48.595228 10501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:48.599187 10501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:48.599349 10501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:48.599566 10501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:48.599679 10501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:48.732462 10501 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:40561
I20240502 14:09:48.732569 10631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:40561 every 8 connection(s)
I20240502 14:09:48.734956 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:48.742465 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10501
I20240502 14:09:48.743094 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:48.744156 28345 external_mini_cluster.cc:1428] Setting key 325e793d3ab26e5164ead59cd0292b8f
May 02 14:09:48 dist-test-slave-dlb8 krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658988, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.66@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:48.766579 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:48.739038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:45459 (local address 127.27.174.126:43251)
0502 14:09:48.740762 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:48.740788 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:48.741873 (+ 1085us) server_negotiation.cc:407] Connection header received
0502 14:09:48.746565 (+ 4692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:48.746584 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:48.746752 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:48.746890 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:48.749460 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:48.750412 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:48.751813 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:48.752381 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:48.756654 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:48.756726 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:48.756744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:48.756844 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:48.760341 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:48.761830 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:48.761874 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:48.761888 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:48.762025 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:48.762874 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:48.762896 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:48.762910 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:48.763318 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:48.763491 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:48.764750 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0502 14:09:48.765075 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":684,"threads_started":1}
WARNING: no policy specified for kudu/127.27.174.67@KRBTEST.COM; defaulting to no policy
I20240502 14:09:48.768061 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:48.739347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43251 (local address 127.27.174.66:45459)
0502 14:09:48.741622 (+ 2275us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:48.741698 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:48.746239 (+ 4541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:48.747202 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:48.747238 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:48.748201 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:48.749198 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:48.749238 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:48.750665 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:48.750688 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:48.751580 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:48.751615 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:48.752003 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:48.752998 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:48.753060 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:48.756369 (+ 3309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:48.760553 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:48.760594 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:48.760636 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:48.761632 (+ 996us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:48.762294 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:48.762332 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:48.762367 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:48.762664 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:48.763545 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:48.763585 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:48.764159 (+ 574us) client_negotiation.cc:769] Sending connection context
0502 14:09:48.764634 (+ 475us) client_negotiation.cc:240] Negotiation successful
0502 14:09:48.765232 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"thread_start_us":1255,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.67@KRBTEST.COM" created.
I20240502 14:09:48.769905 10632 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43251
I20240502 14:09:48.770375 10632 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:48.771782 10632 heartbeater.cc:511] Master 127.27.174.126:43251 requested a full tablet report, sending...
I20240502 14:09:48.774156 10304 ts_manager.cc:194] Registered new tserver with Master: fd2250521136450bbb8c05fd8fbd6a36 (127.27.174.66:40561)
I20240502 14:09:48.775290 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.66@KRBTEST.COM'} at 127.27.174.66:45459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
Entry for principal kudu/127.27.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab.
I20240502 14:09:48.796876 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:49.080361 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:49.080833 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:49.081483 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:49.108994 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:49.109282 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:49.110186 10638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:49.144523 10638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.67.keytab
--principal=kudu/127.27.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:49.145573 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:49.146533 10638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:49.164521 10644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:49.778213 10632 heartbeater.cc:503] Master 127.27.174.126:43251 was elected leader, sending a full tablet report...
W20240502 14:09:49.164795 10647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:49.165356 10645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:49.165719 10638 server_base.cc:1034] running on GCE node
I20240502 14:09:50.185818 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:50.187981 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:50.189267 10638 hybrid_clock.cc:648] HybridClock initialized: now 1714658990189246 us; error 42 us; skew 500 ppm
I20240502 14:09:50.189997 10638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:09:50 dist-test-slave-dlb8 krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:09:50.201496 10638 init.cc:377] Logged in from keytab as kudu/127.27.174.67@KRBTEST.COM (short username kudu)
I20240502 14:09:50.206835 10638 webserver.cc:460] Webserver started at http://127.27.174.67:35867/ using document root <none> and password file <none>
I20240502 14:09:50.207690 10638 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:50.207993 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:50.208439 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:50.213057 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "707c22ff79934c1fb7960956cbba5587"
format_stamp: "Formatted at 2024-05-02 14:09:50 on dist-test-slave-dlb8"
server_key: "19375be05d1190c7ec4558516e205e70"
server_key_iv: "046d215e622e6ff4858f00d8abe90400"
server_key_version: "encryptionkey@0"
I20240502 14:09:50.214141 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "707c22ff79934c1fb7960956cbba5587"
format_stamp: "Formatted at 2024-05-02 14:09:50 on dist-test-slave-dlb8"
server_key: "19375be05d1190c7ec4558516e205e70"
server_key_iv: "046d215e622e6ff4858f00d8abe90400"
server_key_version: "encryptionkey@0"
I20240502 14:09:50.220770 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:09:50.225698 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:50.226627 10638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:09:50.226908 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "707c22ff79934c1fb7960956cbba5587"
format_stamp: "Formatted at 2024-05-02 14:09:50 on dist-test-slave-dlb8"
server_key: "19375be05d1190c7ec4558516e205e70"
server_key_iv: "046d215e622e6ff4858f00d8abe90400"
server_key_version: "encryptionkey@0"
I20240502 14:09:50.227221 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:50.285351 10638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:50.290207 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:50.290614 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:50.293009 10638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:50.296644 10638 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:50.296833 10638 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:50.297037 10638 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:50.297216 10638 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:50.424028 10638 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:45773
I20240502 14:09:50.424131 10767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:45773 every 8 connection(s)
I20240502 14:09:50.426498 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:50.428476 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10638
I20240502 14:09:50.429121 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:50.430979 28345 external_mini_cluster.cc:1428] Setting key 331d71ca773bbaedc66f727b440a745a
May 02 14:09:50 dist-test-slave-dlb8 krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.67@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:09:50.457552 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.431264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:48543 (local address 127.27.174.126:43251)
0502 14:09:50.433013 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.433045 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.434364 (+ 1319us) server_negotiation.cc:407] Connection header received
0502 14:09:50.439105 (+ 4741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.439125 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.439294 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.439425 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:50.441605 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.442687 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.443952 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.444283 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.448095 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:50.448169 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:50.448187 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:50.448287 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:50.451916 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:50.452846 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:50.452870 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:50.452882 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:50.453002 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:50.453587 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:50.453604 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:50.453616 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:50.453955 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:50.454223 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.455049 (+ 826us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.455505 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":599,"threads_started":1}
I20240502 14:09:50.458166 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.431436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:43251 (local address 127.27.174.67:48543)
0502 14:09:50.434120 (+ 2684us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.434203 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.438823 (+ 4620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.439634 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.439657 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.440384 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:50.441296 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.441334 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.442891 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.442914 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.443768 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.443803 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.444060 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.444828 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:09:50.444887 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:50.447814 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:50.452104 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:50.452130 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:50.452149 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:50.452679 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:50.453187 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:09:50.453205 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:09:50.453218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:09:50.453416 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:09:50.454185 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:50.454213 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:50.454559 (+ 346us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.454849 (+ 290us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.455396 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"thread_start_us":1482,"threads_started":1}
I20240502 14:09:50.459793 10768 heartbeater.cc:347] Connected to a master server at 127.27.174.126:43251
I20240502 14:09:50.460201 10768 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:50.461390 10768 heartbeater.cc:511] Master 127.27.174.126:43251 requested a full tablet report, sending...
I20240502 14:09:50.463810 10303 ts_manager.cc:194] Registered new tserver with Master: 707c22ff79934c1fb7960956cbba5587 (127.27.174.67:45773)
I20240502 14:09:50.465057 10303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.174.67@KRBTEST.COM'} at 127.27.174.67:48543
I20240502 14:09:50.477077 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:50.512845 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.493873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.27.174.126:43251)
0502 14:09:50.494723 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.494747 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.495685 (+ 938us) server_negotiation.cc:407] Connection header received
0502 14:09:50.496123 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.496152 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.496378 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.496553 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:50.499155 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.499969 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.501532 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.501958 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.504691 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:50.504786 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:50.504808 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:09:50.504941 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:50.508494 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:50.509393 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:50.509410 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:50.509423 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:50.509567 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:09:50.509967 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:09:50.509983 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:09:50.509995 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:09:50.510490 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:50.510763 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.511278 (+ 515us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.511576 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20240502 14:09:50.518715 10303 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:50.538282 10303 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:50.580123 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.562021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:45773 (local address 127.0.0.1:56334)
0502 14:09:50.568213 (+ 6192us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.568319 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.568736 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.570218 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.570260 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.570391 (+ 131us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:50.571532 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.571603 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.574972 (+ 3369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.575005 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.577062 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.577132 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.577552 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.577704 (+ 152us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.577917 (+ 213us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.578249 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5301,"thread_start_us":15566,"threads_started":1}
I20240502 14:09:50.584009 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.562286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.27.174.67:45773)
0502 14:09:50.567958 (+ 5672us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.568003 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.569036 (+ 1033us) server_negotiation.cc:407] Connection header received
0502 14:09:50.569286 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.569329 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.569743 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.569972 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:50.572422 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.574686 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.579091 (+ 4405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.580272 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.580499 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.580716 (+ 217us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.580935 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4615,"thread_start_us":4156,"threads_started":1}
I20240502 14:09:50.588099 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.563534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:40561 (local address 127.0.0.1:51438)
0502 14:09:50.578456 (+ 14922us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.578654 (+ 198us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.578868 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.580044 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.580070 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.580161 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:50.581049 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.581096 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.583846 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.583895 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.585647 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.585697 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.586023 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.586118 (+ 95us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.586231 (+ 113us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.586440 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14345,"thread_start_us":3512,"threads_started":1}
I20240502 14:09:50.588471 10703 tablet_service.cc:1467] Processing CreateTablet for tablet e3b47b067f8945b49e3a3af69446b526 (DEFAULT_TABLE table=test-table [id=c99a868a00fd4f898dd676856ce17abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:50.589536 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.565620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.27.174.66:40561)
0502 14:09:50.574555 (+ 8935us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.574593 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.578783 (+ 4190us) server_negotiation.cc:407] Connection header received
0502 14:09:50.579109 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.579141 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.579443 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.579639 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:50.581346 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.583474 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.585951 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.587428 (+ 1477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.587662 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.587858 (+ 196us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.588089 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8063,"thread_start_us":14784,"threads_started":1}
I20240502 14:09:50.590615 10703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b47b067f8945b49e3a3af69446b526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:50.593896 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.574663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40085 (local address 127.0.0.1:52394)
0502 14:09:50.581542 (+ 6879us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.581592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.581845 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.587363 (+ 5518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.587412 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.587522 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:50.587993 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.588051 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.590182 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.590207 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.591595 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.591639 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.591922 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.592016 (+ 94us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.592143 (+ 127us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.592367 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6164,"thread_start_us":10051,"threads_started":1}
I20240502 14:09:50.593901 10567 tablet_service.cc:1467] Processing CreateTablet for tablet e3b47b067f8945b49e3a3af69446b526 (DEFAULT_TABLE table=test-table [id=c99a868a00fd4f898dd676856ce17abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:50.595692 10567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b47b067f8945b49e3a3af69446b526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:50.597815 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.573486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.27.174.65:40085)
0502 14:09:50.586082 (+ 12596us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.586113 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.586159 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:09:50.586356 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.586387 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.586697 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.586885 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:50.588378 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.589963 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.594463 (+ 4500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.595732 (+ 1269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.595955 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.596167 (+ 212us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.596396 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11977,"thread_start_us":11689,"threads_started":1}
I20240502 14:09:50.602833 10430 tablet_service.cc:1467] Processing CreateTablet for tablet e3b47b067f8945b49e3a3af69446b526 (DEFAULT_TABLE table=test-table [id=c99a868a00fd4f898dd676856ce17abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:50.604614 10430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b47b067f8945b49e3a3af69446b526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:50.616885 10787 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Bootstrap starting.
I20240502 14:09:50.619520 10788 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Bootstrap starting.
I20240502 14:09:50.623674 10787 tablet_bootstrap.cc:654] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:50.625995 10787 log.cc:824] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:50.626796 10788 tablet_bootstrap.cc:654] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:50.629149 10788 log.cc:824] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:50.632236 10790 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Bootstrap starting.
I20240502 14:09:50.632894 10787 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: No bootstrap required, opened a new log
I20240502 14:09:50.633519 10787 ts_tablet_manager.cc:1367] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240502 14:09:50.637202 10788 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: No bootstrap required, opened a new log
I20240502 14:09:50.637801 10788 ts_tablet_manager.cc:1367] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.006s
I20240502 14:09:50.639147 10790 tablet_bootstrap.cc:654] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:50.640962 10790 log.cc:824] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:50.646209 10790 tablet_bootstrap.cc:492] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: No bootstrap required, opened a new log
I20240502 14:09:50.646595 10790 ts_tablet_manager.cc:1367] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240502 14:09:50.663448 10787 raft_consensus.cc:348] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.664460 10787 raft_consensus.cc:374] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:50.664772 10787 raft_consensus.cc:729] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2250521136450bbb8c05fd8fbd6a36, State: Initialized, Role: FOLLOWER
I20240502 14:09:50.665659 10787 consensus_queue.cc:260] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.668678 10788 raft_consensus.cc:348] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.668785 10790 raft_consensus.cc:348] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.669654 10788 raft_consensus.cc:374] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:50.669729 10787 ts_tablet_manager.cc:1398] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Time spent starting tablet: real 0.036s user 0.031s sys 0.004s
I20240502 14:09:50.669867 10790 raft_consensus.cc:374] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:50.670008 10788 raft_consensus.cc:729] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707c22ff79934c1fb7960956cbba5587, State: Initialized, Role: FOLLOWER
I20240502 14:09:50.670258 10790 raft_consensus.cc:729] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24cb9f7b5a6f49c1be59c57400f8229b, State: Initialized, Role: FOLLOWER
I20240502 14:09:50.671007 10788 consensus_queue.cc:260] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.671128 10790 consensus_queue.cc:260] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.674209 10768 heartbeater.cc:503] Master 127.27.174.126:43251 was elected leader, sending a full tablet report...
I20240502 14:09:50.674775 10788 ts_tablet_manager.cc:1398] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Time spent starting tablet: real 0.037s user 0.032s sys 0.004s
I20240502 14:09:50.679162 10790 ts_tablet_manager.cc:1398] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Time spent starting tablet: real 0.032s user 0.027s sys 0.004s
W20240502 14:09:50.681385 10769 tablet.cc:2369] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:50.740061 10793 raft_consensus.cc:482] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:50.740510 10793 raft_consensus.cc:504] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
W20240502 14:09:50.741485 10633 tablet.cc:2369] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:50.743362 10793 leader_election.cc:288] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 707c22ff79934c1fb7960956cbba5587 (127.27.174.67:45773), 24cb9f7b5a6f49c1be59c57400f8229b (127.27.174.65:40085)
I20240502 14:09:50.755162 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.743700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:45773 (local address 127.27.174.66:57759)
0502 14:09:50.746537 (+ 2837us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.746595 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.746987 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.747858 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.747886 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.748001 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:50.749083 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.749129 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.750938 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.750974 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.752385 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.752433 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.752820 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.752925 (+ 105us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.753093 (+ 168us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.753338 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2012,"thread_start_us":2518,"threads_started":1}
I20240502 14:09:50.755591 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.743926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57759 (local address 127.27.174.67:45773)
0502 14:09:50.745645 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.745673 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.746706 (+ 1033us) server_negotiation.cc:407] Connection header received
0502 14:09:50.747344 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.747375 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.747645 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.747805 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:50.749374 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.750662 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.752669 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.753783 (+ 1114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.753920 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.754123 (+ 203us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.754339 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105}
I20240502 14:09:50.757135 10723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b47b067f8945b49e3a3af69446b526" candidate_uuid: "fd2250521136450bbb8c05fd8fbd6a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707c22ff79934c1fb7960956cbba5587" is_pre_election: true
I20240502 14:09:50.757802 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.744448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:52947 (local address 127.27.174.65:40085)
0502 14:09:50.745342 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.745367 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.747693 (+ 2326us) server_negotiation.cc:407] Connection header received
0502 14:09:50.748107 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.748141 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.748410 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.748585 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:50.750189 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.751055 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.753996 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.755194 (+ 1198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.755336 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.756261 (+ 925us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.756484 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20240502 14:09:50.758102 10723 raft_consensus.cc:2454] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd2250521136450bbb8c05fd8fbd6a36 in term 0.
I20240502 14:09:50.759011 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.743996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40085 (local address 127.27.174.66:52947)
0502 14:09:50.747521 (+ 3525us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.747566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:50.747812 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:50.748989 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:50.749015 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:50.749117 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:50.749926 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.749960 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.751313 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.751341 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:50.753659 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:50.753728 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.755589 (+ 1861us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.755969 (+ 380us) client_negotiation.cc:769] Sending connection context
0502 14:09:50.756866 (+ 897us) client_negotiation.cc:240] Negotiation successful
0502 14:09:50.757064 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2961,"thread_start_us":2592,"threads_started":1}
I20240502 14:09:50.759469 10520 leader_election.cc:302] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707c22ff79934c1fb7960956cbba5587, fd2250521136450bbb8c05fd8fbd6a36; no voters:
I20240502 14:09:50.760202 10793 raft_consensus.cc:2781] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:50.760564 10793 raft_consensus.cc:482] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:50.760582 10450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b47b067f8945b49e3a3af69446b526" candidate_uuid: "fd2250521136450bbb8c05fd8fbd6a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" is_pre_election: true
I20240502 14:09:50.760865 10793 raft_consensus.cc:3037] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:50.761355 10450 raft_consensus.cc:2454] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd2250521136450bbb8c05fd8fbd6a36 in term 0.
I20240502 14:09:50.765956 10793 raft_consensus.cc:504] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.767303 10793 leader_election.cc:288] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [CANDIDATE]: Term 1 election: Requested vote from peers 707c22ff79934c1fb7960956cbba5587 (127.27.174.67:45773), 24cb9f7b5a6f49c1be59c57400f8229b (127.27.174.65:40085)
I20240502 14:09:50.768158 10723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b47b067f8945b49e3a3af69446b526" candidate_uuid: "fd2250521136450bbb8c05fd8fbd6a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707c22ff79934c1fb7960956cbba5587"
I20240502 14:09:50.768324 10450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b47b067f8945b49e3a3af69446b526" candidate_uuid: "fd2250521136450bbb8c05fd8fbd6a36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24cb9f7b5a6f49c1be59c57400f8229b"
I20240502 14:09:50.768573 10723 raft_consensus.cc:3037] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:50.768833 10450 raft_consensus.cc:3037] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:50.773140 10723 raft_consensus.cc:2454] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2250521136450bbb8c05fd8fbd6a36 in term 1.
I20240502 14:09:50.773964 10520 leader_election.cc:302] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707c22ff79934c1fb7960956cbba5587, fd2250521136450bbb8c05fd8fbd6a36; no voters:
I20240502 14:09:50.773670 10450 raft_consensus.cc:2454] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2250521136450bbb8c05fd8fbd6a36 in term 1.
I20240502 14:09:50.774631 10793 raft_consensus.cc:2781] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:50.776048 10793 raft_consensus.cc:686] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 1 LEADER]: Becoming Leader. State: Replica: fd2250521136450bbb8c05fd8fbd6a36, State: Running, Role: LEADER
I20240502 14:09:50.776711 10793 consensus_queue.cc:237] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } }
I20240502 14:09:50.787550 10302 catalog_manager.cc:5484] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd2250521136450bbb8c05fd8fbd6a36 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "fd2250521136450bbb8c05fd8fbd6a36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd2250521136450bbb8c05fd8fbd6a36" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 } health_report { overall_health: UNKNOWN } } }
W20240502 14:09:50.824069 10496 tablet.cc:2369] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:50.836761 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.830779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.27.174.66:40561)
0502 14:09:50.831285 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:50.831302 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:50.831336 (+ 34us) server_negotiation.cc:407] Connection header received
0502 14:09:50.831640 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:50.831656 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:50.831858 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:50.831991 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:50.832618 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.833315 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:50.834515 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:50.834830 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:50.834979 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:50.835451 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:50.835574 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:50.835715 (+ 141us) server_negotiation.cc:299] Negotiation successful
0502 14:09:50.835817 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20240502 14:09:50.848146 10450 raft_consensus.cc:1261] T e3b47b067f8945b49e3a3af69446b526 P 24cb9f7b5a6f49c1be59c57400f8229b [term 1 FOLLOWER]: Refusing update from remote peer fd2250521136450bbb8c05fd8fbd6a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:50.848687 10723 raft_consensus.cc:1261] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 1 FOLLOWER]: Refusing update from remote peer fd2250521136450bbb8c05fd8fbd6a36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:50.849807 10798 consensus_queue.cc:1035] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24cb9f7b5a6f49c1be59c57400f8229b" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:50.850411 10793 consensus_queue.cc:1035] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "707c22ff79934c1fb7960956cbba5587" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:50.877571 10803 mvcc.cc:205] Tried to move back new op lower bound from 7023243226499993600 to 7023243226228117504. Current Snapshot: MvccSnapshot[applied={T|T < 7023243226499993600}]
I20240502 14:09:50.897133 10304 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47838:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:50.897645 10304 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47838:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:50.906289 10304 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 373972bd046a4115a200069910bad2af: Sending DeleteTablet for 3 replicas of tablet e3b47b067f8945b49e3a3af69446b526
I20240502 14:09:50.908149 10703 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b47b067f8945b49e3a3af69446b526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:50 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:56334
I20240502 14:09:50.909596 10430 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b47b067f8945b49e3a3af69446b526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:50 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:52394
I20240502 14:09:50.909181 10567 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b47b067f8945b49e3a3af69446b526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:50 UTC) from {username='kudu', principal='kudu/127.27.174.126@KRBTEST.COM'} at 127.0.0.1:51438
I20240502 14:09:50.915417 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10365
I20240502 14:09:50.917533 10813 tablet_replica.cc:330] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: stopping tablet replica
I20240502 14:09:50.918061 10811 tablet_replica.cc:330] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: stopping tablet replica
I20240502 14:09:50.918406 10813 raft_consensus.cc:2229] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:50.918947 10811 raft_consensus.cc:2229] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:50.919543 10813 raft_consensus.cc:2258] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:50.919574 10811 raft_consensus.cc:2258] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:50.922878 10813 ts_tablet_manager.cc:1875] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:50.922940 10811 ts_tablet_manager.cc:1875] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:50.937088 10813 ts_tablet_manager.cc:1888] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:50.937623 10813 log.cc:1196] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/e3b47b067f8945b49e3a3af69446b526
I20240502 14:09:50.938524 10813 ts_tablet_manager.cc:1909] T e3b47b067f8945b49e3a3af69446b526 P fd2250521136450bbb8c05fd8fbd6a36: Deleting consensus metadata
I20240502 14:09:50.940846 10811 ts_tablet_manager.cc:1888] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:50.941304 10811 log.cc:1196] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/e3b47b067f8945b49e3a3af69446b526
I20240502 14:09:50.941709 10289 catalog_manager.cc:4830] TS fd2250521136450bbb8c05fd8fbd6a36 (127.27.174.66:40561): tablet e3b47b067f8945b49e3a3af69446b526 (table test-table [id=c99a868a00fd4f898dd676856ce17abc]) successfully deleted
I20240502 14:09:50.942138 10811 ts_tablet_manager.cc:1909] T e3b47b067f8945b49e3a3af69446b526 P 707c22ff79934c1fb7960956cbba5587: Deleting consensus metadata
I20240502 14:09:50.945183 10288 catalog_manager.cc:4830] TS 707c22ff79934c1fb7960956cbba5587 (127.27.174.67:45773): tablet e3b47b067f8945b49e3a3af69446b526 (table test-table [id=c99a868a00fd4f898dd676856ce17abc]) successfully deleted
W20240502 14:09:50.950135 10289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:40085 (error 108)
I20240502 14:09:50.951182 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10501
I20240502 14:09:50.955761 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:50.954450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40085 (local address 127.0.0.1:52400)
0502 14:09:50.955053 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:50.955259 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20240502 14:09:50.956279 10289 catalog_manager.cc:4573] TS 24cb9f7b5a6f49c1be59c57400f8229b (127.27.174.65:40085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3b47b067f8945b49e3a3af69446b526: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40085: connect: Connection refused (error 111)
I20240502 14:09:50.974735 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10638
I20240502 14:09:50.997623 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10270
2024-05-02T14:09:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (8561 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-02T14:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:51Z Disabled control of system clock
I20240502 14:09:51.081848 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40861
--webserver_interface=127.27.174.126
--webserver_port=0
--builtin_ntp_servers=127.27.174.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:51.361268 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:51.361935 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:51.362509 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:51.392097 10818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:51.392450 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:51.392668 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:51.392922 10818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:51.393146 10818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:51.427367 10818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:40861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:51.428654 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:51.430174 10818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:51.444379 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:51.444707 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:51.445418 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:51.446125 10818 server_base.cc:1034] running on GCE node
I20240502 14:09:52.472895 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:52.475422 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:52.476759 10818 hybrid_clock.cc:648] HybridClock initialized: now 1714658992476723 us; error 58 us; skew 500 ppm
I20240502 14:09:52.477526 10818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:52.483323 10818 webserver.cc:460] Webserver started at http://127.27.174.126:44043/ using document root <none> and password file <none>
I20240502 14:09:52.484139 10818 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:52.484342 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:52.484748 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:52.489667 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "e22b3101e315498f94affcc8bbbbf207"
format_stamp: "Formatted at 2024-05-02 14:09:52 on dist-test-slave-dlb8"
server_key: "6805db5f9c778a9ea70652180cbf66c4"
server_key_iv: "59d747c108e9c217464f794cf1395621"
server_key_version: "encryptionkey@0"
I20240502 14:09:52.490751 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "e22b3101e315498f94affcc8bbbbf207"
format_stamp: "Formatted at 2024-05-02 14:09:52 on dist-test-slave-dlb8"
server_key: "6805db5f9c778a9ea70652180cbf66c4"
server_key_iv: "59d747c108e9c217464f794cf1395621"
server_key_version: "encryptionkey@0"
I20240502 14:09:52.497135 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:09:52.502379 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:52.503319 10818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:09:52.503607 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "e22b3101e315498f94affcc8bbbbf207"
format_stamp: "Formatted at 2024-05-02 14:09:52 on dist-test-slave-dlb8"
server_key: "6805db5f9c778a9ea70652180cbf66c4"
server_key_iv: "59d747c108e9c217464f794cf1395621"
server_key_version: "encryptionkey@0"
I20240502 14:09:52.503932 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:52.552143 10818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:52.553516 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:52.553946 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:52.618956 10818 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:40861
I20240502 14:09:52.619055 10886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:40861 every 8 connection(s)
I20240502 14:09:52.621467 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:09:52.626761 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:52.631474 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10818
I20240502 14:09:52.631888 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:09:52.632881 28345 external_mini_cluster.cc:1428] Setting key 422ff175b65da0b48d2c783226954cee
I20240502 14:09:52.651448 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: Bootstrap starting.
I20240502 14:09:52.655206 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:52.635016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.27.174.126:40861)
0502 14:09:52.638142 (+ 3126us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:52.638183 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:52.638259 (+ 76us) server_negotiation.cc:407] Connection header received
0502 14:09:52.643526 (+ 5267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:52.643584 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:52.644098 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:52.644544 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:52.645840 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:52.648058 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:52.649480 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:52.650144 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:52.650704 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:52.650825 (+ 121us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:52.650864 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:52.651907 (+ 1043us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:52.652030 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:52.652086 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:52.652284 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:52.652535 (+ 251us) server_negotiation.cc:299] Negotiation successful
0502 14:09:52.652836 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2221,"thread_start_us":1849,"threads_started":1}
I20240502 14:09:52.658212 10887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:52.659951 10887 log.cc:824] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:52.664693 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: No bootstrap required, opened a new log
I20240502 14:09:52.684710 10887 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } }
I20240502 14:09:52.685580 10887 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:52.685873 10887 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22b3101e315498f94affcc8bbbbf207, State: Initialized, Role: FOLLOWER
I20240502 14:09:52.686779 10887 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } }
I20240502 14:09:52.687549 10887 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:09:52.687907 10887 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:09:52.688352 10887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:52.693686 10887 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } }
I20240502 14:09:52.694402 10887 leader_election.cc:302] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22b3101e315498f94affcc8bbbbf207; no voters:
I20240502 14:09:52.696205 10887 leader_election.cc:288] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:09:52.697095 10892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:52.699523 10892 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [term 1 LEADER]: Becoming Leader. State: Replica: e22b3101e315498f94affcc8bbbbf207, State: Running, Role: LEADER
I20240502 14:09:52.700194 10892 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } }
I20240502 14:09:52.700606 10887 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:09:52.714618 10894 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22b3101e315498f94affcc8bbbbf207. Latest consensus state: current_term: 1 leader_uuid: "e22b3101e315498f94affcc8bbbbf207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } } }
I20240502 14:09:52.715947 10894 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:52.714604 10893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22b3101e315498f94affcc8bbbbf207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22b3101e315498f94affcc8bbbbf207" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 40861 } } }
I20240502 14:09:52.717855 10893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207 [sys.catalog]: This master's current role is: LEADER
I20240502 14:09:52.725201 10901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:09:52.735373 10901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:09:52.751430 10901 catalog_manager.cc:1347] Generated new cluster ID: bca53c6d132743bb9ecb6a6ac34855d1
I20240502 14:09:52.751755 10901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:09:52.766842 10901 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:09:52.768251 10901 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:09:52.784523 10901 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: Generated new TSK 0
I20240502 14:09:52.785308 10901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:09:52.794535 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.65:0
--local_ip_for_outbound_sockets=127.27.174.65
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--builtin_ntp_servers=127.27.174.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240502 14:09:53.084530 10911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:53.085013 10911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:53.085501 10911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:53.112958 10911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:53.113174 10911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:53.113993 10911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.65
I20240502 14:09:53.145216 10911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.174.65
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.65
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:53.146348 10911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:53.147778 10911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:53.165030 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:53.165505 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:53.166635 10911 server_base.cc:1034] running on GCE node
W20240502 14:09:53.166162 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:54.215562 10911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:54.218333 10911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:54.219810 10911 hybrid_clock.cc:648] HybridClock initialized: now 1714658994219758 us; error 55 us; skew 500 ppm
I20240502 14:09:54.220772 10911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:54.227994 10911 webserver.cc:460] Webserver started at http://127.27.174.65:34139/ using document root <none> and password file <none>
I20240502 14:09:54.229240 10911 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:54.229494 10911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:54.230082 10911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:54.237170 10911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data/instance:
uuid: "53c75e0336c34ab68fdc9e7e005bda2e"
format_stamp: "Formatted at 2024-05-02 14:09:54 on dist-test-slave-dlb8"
server_key: "9e871a26c2a6beab63c4b87d26e5ecb5"
server_key_iv: "de605dfa8a03cddb2bb80f815d71deb5"
server_key_version: "encryptionkey@0"
I20240502 14:09:54.238612 10911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance:
uuid: "53c75e0336c34ab68fdc9e7e005bda2e"
format_stamp: "Formatted at 2024-05-02 14:09:54 on dist-test-slave-dlb8"
server_key: "9e871a26c2a6beab63c4b87d26e5ecb5"
server_key_iv: "de605dfa8a03cddb2bb80f815d71deb5"
server_key_version: "encryptionkey@0"
I20240502 14:09:54.245510 10911 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240502 14:09:54.251166 10927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:54.252147 10911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:09:54.252450 10911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal
uuid: "53c75e0336c34ab68fdc9e7e005bda2e"
format_stamp: "Formatted at 2024-05-02 14:09:54 on dist-test-slave-dlb8"
server_key: "9e871a26c2a6beab63c4b87d26e5ecb5"
server_key_iv: "de605dfa8a03cddb2bb80f815d71deb5"
server_key_version: "encryptionkey@0"
I20240502 14:09:54.252789 10911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:54.298476 10911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:54.299827 10911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:54.300263 10911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:54.303138 10911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:54.307015 10911 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:54.307191 10911 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:54.307438 10911 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:54.307585 10911 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:54.462985 10911 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.65:40511
I20240502 14:09:54.463083 11039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.65:40511 every 8 connection(s)
I20240502 14:09:54.465188 10911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/data/info.pb
I20240502 14:09:54.469698 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 10911
I20240502 14:09:54.470218 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-0/wal/instance
I20240502 14:09:54.471351 28345 external_mini_cluster.cc:1428] Setting key b4ad300ce88c948149ee92570ccfc69f
I20240502 14:09:54.475996 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.66:0
--local_ip_for_outbound_sockets=127.27.174.66
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--builtin_ntp_servers=127.27.174.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:54.485700 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:54.469048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.65:45693 (local address 127.27.174.126:40861)
0502 14:09:54.470817 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:54.470843 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:54.472434 (+ 1591us) server_negotiation.cc:407] Connection header received
0502 14:09:54.477021 (+ 4587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:54.477053 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:54.477304 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:54.477498 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:54.479113 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:54.480655 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:54.481824 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:54.482491 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:54.483120 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:54.483190 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:54.483205 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:54.483241 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:54.483309 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:54.483341 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:54.483438 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:54.483777 (+ 339us) server_negotiation.cc:299] Negotiation successful
0502 14:09:54.483911 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":605,"threads_started":1}
I20240502 14:09:54.485785 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:54.469359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40861 (local address 127.27.174.65:45693)
0502 14:09:54.472214 (+ 2855us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:54.472275 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:54.476708 (+ 4433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:54.477661 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:54.477682 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:54.477822 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:54.478835 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:54.478878 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:54.480871 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:54.480888 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:54.481602 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:54.481633 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:54.481910 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:54.482749 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:54.482764 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:54.482817 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:54.482918 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:54.483510 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:54.483544 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:54.483573 (+ 29us) client_negotiation.cc:769] Sending connection context
0502 14:09:54.483751 (+ 178us) client_negotiation.cc:240] Negotiation successful
0502 14:09:54.484056 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"thread_start_us":1605,"threads_started":1}
I20240502 14:09:54.487730 11040 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40861
I20240502 14:09:54.488121 11040 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:54.489079 11040 heartbeater.cc:511] Master 127.27.174.126:40861 requested a full tablet report, sending...
I20240502 14:09:54.491400 10851 ts_manager.cc:194] Registered new tserver with Master: 53c75e0336c34ab68fdc9e7e005bda2e (127.27.174.65:40511)
I20240502 14:09:54.493114 10851 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.65:45693
W20240502 14:09:54.766480 11044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:54.766942 11044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:54.767431 11044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:54.794961 11044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:54.795234 11044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:54.796006 11044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.66
I20240502 14:09:54.827736 11044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.174.66
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.66
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:54.828948 11044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:54.830446 11044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:54.845350 11051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:55.496111 11040 heartbeater.cc:503] Master 127.27.174.126:40861 was elected leader, sending a full tablet report...
W20240502 14:09:54.846511 11053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:54.848438 11044 server_base.cc:1034] running on GCE node
W20240502 14:09:54.846122 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:55.859969 11044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:55.862262 11044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:55.863602 11044 hybrid_clock.cc:648] HybridClock initialized: now 1714658995863551 us; error 71 us; skew 500 ppm
I20240502 14:09:55.864393 11044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:55.875722 11044 webserver.cc:460] Webserver started at http://127.27.174.66:35745/ using document root <none> and password file <none>
I20240502 14:09:55.876582 11044 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:55.876770 11044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:55.877163 11044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:55.882031 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data/instance:
uuid: "8e83cc6b4d4f4ab08f64bd47f7399002"
format_stamp: "Formatted at 2024-05-02 14:09:55 on dist-test-slave-dlb8"
server_key: "cef06da553a9f2e06c39b5f09a5f3ca0"
server_key_iv: "d60eb40364eeb84629895e021f9d1567"
server_key_version: "encryptionkey@0"
I20240502 14:09:55.883118 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance:
uuid: "8e83cc6b4d4f4ab08f64bd47f7399002"
format_stamp: "Formatted at 2024-05-02 14:09:55 on dist-test-slave-dlb8"
server_key: "cef06da553a9f2e06c39b5f09a5f3ca0"
server_key_iv: "d60eb40364eeb84629895e021f9d1567"
server_key_version: "encryptionkey@0"
I20240502 14:09:55.889569 11044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:09:55.894317 11060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:55.895208 11044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:09:55.895474 11044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal
uuid: "8e83cc6b4d4f4ab08f64bd47f7399002"
format_stamp: "Formatted at 2024-05-02 14:09:55 on dist-test-slave-dlb8"
server_key: "cef06da553a9f2e06c39b5f09a5f3ca0"
server_key_iv: "d60eb40364eeb84629895e021f9d1567"
server_key_version: "encryptionkey@0"
I20240502 14:09:55.895794 11044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:55.941819 11044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:55.943255 11044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:55.943645 11044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:55.945894 11044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:55.949682 11044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:55.949859 11044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:55.950079 11044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:55.950235 11044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:56.073370 11044 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.66:45929
I20240502 14:09:56.073459 11172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.66:45929 every 8 connection(s)
I20240502 14:09:56.075700 11044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/data/info.pb
I20240502 14:09:56.080221 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11044
I20240502 14:09:56.080739 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal/instance
I20240502 14:09:56.081856 28345 external_mini_cluster.cc:1428] Setting key e4da478f7983d8ca46139fdab075168a
I20240502 14:09:56.086686 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.174.67:0
--local_ip_for_outbound_sockets=127.27.174.67
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--builtin_ntp_servers=127.27.174.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240502 14:09:56.095937 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:56.079519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:38845 (local address 127.27.174.126:40861)
0502 14:09:56.081047 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:56.081074 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:56.082960 (+ 1886us) server_negotiation.cc:407] Connection header received
0502 14:09:56.087536 (+ 4576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:56.087566 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:56.087837 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:56.088056 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:56.089618 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:56.090858 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:56.092137 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:56.092695 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:56.093360 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:56.093426 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:56.093444 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:56.093481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:56.093556 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:56.093589 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:56.093707 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:56.093993 (+ 286us) server_negotiation.cc:299] Negotiation successful
0502 14:09:56.094198 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":546,"threads_started":1}
I20240502 14:09:56.096640 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:56.079830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40861 (local address 127.27.174.66:38845)
0502 14:09:56.082743 (+ 2913us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:56.082809 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:56.087225 (+ 4416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:56.088255 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:56.088287 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:56.088411 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:56.089359 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:56.089396 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:56.091091 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:56.091109 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:56.091904 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:56.091941 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:56.092214 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:56.092979 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:56.092994 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:56.093044 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:56.093137 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:56.093746 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:56.093772 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:56.093789 (+ 17us) client_negotiation.cc:769] Sending connection context
0502 14:09:56.093946 (+ 157us) client_negotiation.cc:240] Negotiation successful
0502 14:09:56.094242 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"thread_start_us":1692,"threads_started":1}
I20240502 14:09:56.098506 11173 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40861
I20240502 14:09:56.098886 11173 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:56.099817 11173 heartbeater.cc:511] Master 127.27.174.126:40861 requested a full tablet report, sending...
I20240502 14:09:56.101845 10851 ts_manager.cc:194] Registered new tserver with Master: 8e83cc6b4d4f4ab08f64bd47f7399002 (127.27.174.66:45929)
I20240502 14:09:56.103066 10851 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.66:38845
W20240502 14:09:56.373216 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:56.373661 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:56.374177 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:56.403105 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:56.403390 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:56.404196 11177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.174.67
I20240502 14:09:56.438042 11177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.174.67
--webserver_port=0
--tserver_master_addrs=127.27.174.126:40861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.174.67
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:56.439244 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:56.440800 11177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:56.455642 11183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:57.105803 11173 heartbeater.cc:503] Master 127.27.174.126:40861 was elected leader, sending a full tablet report...
W20240502 14:09:56.456308 11186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:56.456979 11184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:56.457506 11177 server_base.cc:1034] running on GCE node
I20240502 14:09:57.476640 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:09:57.479480 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:09:57.480903 11177 hybrid_clock.cc:648] HybridClock initialized: now 1714658997480880 us; error 33 us; skew 500 ppm
I20240502 14:09:57.481685 11177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:09:57.488376 11177 webserver.cc:460] Webserver started at http://127.27.174.67:37207/ using document root <none> and password file <none>
I20240502 14:09:57.489254 11177 fs_manager.cc:359] Metadata directory not provided
I20240502 14:09:57.489461 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:09:57.489858 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:09:57.494650 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data/instance:
uuid: "9263e6cd5dee484ab8dfddc94a52a772"
format_stamp: "Formatted at 2024-05-02 14:09:57 on dist-test-slave-dlb8"
server_key: "24034ecd2e23f77f9225f2af74f04dd4"
server_key_iv: "de1f5591d2b6d1aa778dab8c51552e99"
server_key_version: "encryptionkey@0"
I20240502 14:09:57.495688 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance:
uuid: "9263e6cd5dee484ab8dfddc94a52a772"
format_stamp: "Formatted at 2024-05-02 14:09:57 on dist-test-slave-dlb8"
server_key: "24034ecd2e23f77f9225f2af74f04dd4"
server_key_iv: "de1f5591d2b6d1aa778dab8c51552e99"
server_key_version: "encryptionkey@0"
I20240502 14:09:57.502471 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:09:57.507848 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:57.509052 11177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20240502 14:09:57.509387 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal
uuid: "9263e6cd5dee484ab8dfddc94a52a772"
format_stamp: "Formatted at 2024-05-02 14:09:57 on dist-test-slave-dlb8"
server_key: "24034ecd2e23f77f9225f2af74f04dd4"
server_key_iv: "de1f5591d2b6d1aa778dab8c51552e99"
server_key_version: "encryptionkey@0"
I20240502 14:09:57.509724 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:09:57.564707 11177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:09:57.566150 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:09:57.566566 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:09:57.569020 11177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:09:57.572901 11177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:09:57.573086 11177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:57.573349 11177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:09:57.573486 11177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:09:57.708089 11177 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.67:42119
I20240502 14:09:57.708206 11305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.67:42119 every 8 connection(s)
I20240502 14:09:57.710567 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/data/info.pb
I20240502 14:09:57.720712 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11177
I20240502 14:09:57.721339 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal/instance
I20240502 14:09:57.722503 28345 external_mini_cluster.cc:1428] Setting key 0e2964e70409dd55b80fd8855eda67fe
I20240502 14:09:57.732291 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.714520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.67:32987 (local address 127.27.174.126:40861)
0502 14:09:57.716101 (+ 1581us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.716132 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.718283 (+ 2151us) server_negotiation.cc:407] Connection header received
0502 14:09:57.722978 (+ 4695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.723009 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.723267 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.723471 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:57.725277 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.726590 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.728020 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.728607 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.729558 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:57.729630 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:57.729647 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:57.729690 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:57.729759 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:57.729794 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:57.729919 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.730272 (+ 353us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.730469 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":524,"threads_started":1}
I20240502 14:09:57.732975 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.714881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:40861 (local address 127.27.174.67:32987)
0502 14:09:57.717977 (+ 3096us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.718042 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.722599 (+ 4557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.723743 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.723766 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.723882 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:09:57.725011 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.725057 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.726838 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.726858 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.727786 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.727824 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.728201 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.729140 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 14:09:57.729155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:09:57.729209 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 14:09:57.729327 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:09:57.729959 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:09:57.729986 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:09:57.730004 (+ 18us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.730210 (+ 206us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.730520 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2258,"thread_start_us":1870,"threads_started":1}
I20240502 14:09:57.734526 11306 heartbeater.cc:347] Connected to a master server at 127.27.174.126:40861
I20240502 14:09:57.734885 11306 heartbeater.cc:464] Registering TS with master...
I20240502 14:09:57.735875 11306 heartbeater.cc:511] Master 127.27.174.126:40861 requested a full tablet report, sending...
I20240502 14:09:57.737871 10851 ts_manager.cc:194] Registered new tserver with Master: 9263e6cd5dee484ab8dfddc94a52a772 (127.27.174.67:42119)
I20240502 14:09:57.739210 10851 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.174.67:32987
I20240502 14:09:57.742201 28345 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 14:09:57.766170 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.757292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.27.174.126:40861)
0502 14:09:57.758172 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.758199 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.758994 (+ 795us) server_negotiation.cc:407] Connection header received
0502 14:09:57.759341 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.759357 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.759510 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.759635 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:09:57.760744 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.761553 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.762871 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.763206 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.764211 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:09:57.764291 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:09:57.764307 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:09:57.764355 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:09:57.764445 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:09:57.764499 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:09:57.764650 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.764882 (+ 232us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.765077 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20240502 14:09:57.771375 10851 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:09:57.790699 10851 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:09:57.830008 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.813843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.66:45929 (local address 127.0.0.1:36944)
0502 14:09:57.816173 (+ 2330us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.816259 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.816639 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.818463 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.818514 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.818667 (+ 153us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:57.819681 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.819749 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.823507 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.823548 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.826895 (+ 3347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.826967 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.827404 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.827554 (+ 150us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.827748 (+ 194us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.828033 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":3002,"threads_started":1}
I20240502 14:09:57.832945 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.814198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.27.174.66:45929)
0502 14:09:57.816947 (+ 2749us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.816978 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.817033 (+ 55us) server_negotiation.cc:407] Connection header received
0502 14:09:57.817225 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.817270 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.817629 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.817859 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:57.820094 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.823093 (+ 2999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.829694 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.830833 (+ 1139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.831144 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.831345 (+ 201us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.831610 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2161,"thread_start_us":1802,"threads_started":1}
I20240502 14:09:57.837419 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.815569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40511 (local address 127.0.0.1:35100)
0502 14:09:57.819092 (+ 3523us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.819349 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.819567 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.825167 (+ 5600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.825205 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.825359 (+ 154us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:57.827956 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.828020 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.833587 (+ 5567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.833607 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.835075 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.835124 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.835456 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.835553 (+ 97us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.835692 (+ 139us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.835890 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2920,"thread_start_us":4053,"threads_started":1}
I20240502 14:09:57.837664 11108 tablet_service.cc:1467] Processing CreateTablet for tablet 83332578555d434e93af9bd0fb059fec (DEFAULT_TABLE table=test-table [id=7636ddadd7db44b195436fa7c86d2740]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:57.839648 11108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83332578555d434e93af9bd0fb059fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:57.840109 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.820724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42119 (local address 127.0.0.1:51712)
0502 14:09:57.830481 (+ 9757us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.830549 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.830775 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.832142 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.832162 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.832228 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:57.832886 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.832916 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.835265 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.835294 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.836972 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.837018 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.837334 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.837420 (+ 86us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.837565 (+ 145us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.838457 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9200,"thread_start_us":8501,"threads_started":1}
I20240502 14:09:57.841048 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.816878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.27.174.65:40511)
0502 14:09:57.820958 (+ 4080us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.820994 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.821053 (+ 59us) server_negotiation.cc:407] Connection header received
0502 14:09:57.821282 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.821318 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.821697 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.821959 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:57.831142 (+ 9183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.833338 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.835342 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.838905 (+ 3563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.839222 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.839458 (+ 236us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.839693 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3488,"thread_start_us":3180,"threads_started":1}
I20240502 14:09:57.843744 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.820728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.27.174.67:42119)
0502 14:09:57.829883 (+ 9155us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.829914 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.830967 (+ 1053us) server_negotiation.cc:407] Connection header received
0502 14:09:57.831152 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.831183 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.831486 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.831680 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:57.833147 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.835043 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.837958 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.839240 (+ 1282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.839556 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.839777 (+ 221us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.839994 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8405,"thread_start_us":7938,"threads_started":1}
I20240502 14:09:57.845026 10975 tablet_service.cc:1467] Processing CreateTablet for tablet 83332578555d434e93af9bd0fb059fec (DEFAULT_TABLE table=test-table [id=7636ddadd7db44b195436fa7c86d2740]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:57.846730 10975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83332578555d434e93af9bd0fb059fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:57.847062 11241 tablet_service.cc:1467] Processing CreateTablet for tablet 83332578555d434e93af9bd0fb059fec (DEFAULT_TABLE table=test-table [id=7636ddadd7db44b195436fa7c86d2740]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:09:57.848371 11241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83332578555d434e93af9bd0fb059fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:09:57.866621 11326 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Bootstrap starting.
I20240502 14:09:57.869599 11327 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Bootstrap starting.
I20240502 14:09:57.872535 11326 tablet_bootstrap.cc:654] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:57.874437 11328 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Bootstrap starting.
I20240502 14:09:57.876257 11327 tablet_bootstrap.cc:654] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:57.876967 11326 log.cc:824] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:57.878504 11327 log.cc:824] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:57.879904 11328 tablet_bootstrap.cc:654] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Neither blocks nor log segments found. Creating new log.
I20240502 14:09:57.882117 11328 log.cc:824] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:09:57.882997 11326 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: No bootstrap required, opened a new log
I20240502 14:09:57.883491 11326 ts_tablet_manager.cc:1367] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240502 14:09:57.883932 11327 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: No bootstrap required, opened a new log
I20240502 14:09:57.884474 11327 ts_tablet_manager.cc:1367] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240502 14:09:57.887723 11328 tablet_bootstrap.cc:492] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: No bootstrap required, opened a new log
I20240502 14:09:57.888239 11328 ts_tablet_manager.cc:1367] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 14:09:57.903770 11327 raft_consensus.cc:348] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.904393 11327 raft_consensus.cc:374] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:57.904642 11327 raft_consensus.cc:729] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9263e6cd5dee484ab8dfddc94a52a772, State: Initialized, Role: FOLLOWER
I20240502 14:09:57.905467 11327 consensus_queue.cc:260] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.908973 11306 heartbeater.cc:503] Master 127.27.174.126:40861 was elected leader, sending a full tablet report...
I20240502 14:09:57.910203 11327 ts_tablet_manager.cc:1398] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Time spent starting tablet: real 0.025s user 0.025s sys 0.000s
I20240502 14:09:57.912848 11326 raft_consensus.cc:348] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.913439 11326 raft_consensus.cc:374] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:57.913658 11326 raft_consensus.cc:729] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e83cc6b4d4f4ab08f64bd47f7399002, State: Initialized, Role: FOLLOWER
I20240502 14:09:57.914340 11326 consensus_queue.cc:260] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.916582 11328 raft_consensus.cc:348] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.917541 11328 raft_consensus.cc:374] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:09:57.917867 11328 raft_consensus.cc:729] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c75e0336c34ab68fdc9e7e005bda2e, State: Initialized, Role: FOLLOWER
I20240502 14:09:57.918825 11326 ts_tablet_manager.cc:1398] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Time spent starting tablet: real 0.035s user 0.021s sys 0.012s
I20240502 14:09:57.919212 11328 consensus_queue.cc:260] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.923157 11328 ts_tablet_manager.cc:1398] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Time spent starting tablet: real 0.035s user 0.025s sys 0.008s
W20240502 14:09:57.964888 11307 tablet.cc:2369] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:57.971465 11333 raft_consensus.cc:482] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:09:57.971884 11333 raft_consensus.cc:504] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
W20240502 14:09:57.973110 11041 tablet.cc:2369] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:09:57.974061 11333 leader_election.cc:288] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c75e0336c34ab68fdc9e7e005bda2e (127.27.174.65:40511), 9263e6cd5dee484ab8dfddc94a52a772 (127.27.174.67:42119)
I20240502 14:09:57.984665 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.974511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:57455 (local address 127.27.174.65:40511)
0502 14:09:57.975242 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.975282 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.976498 (+ 1216us) server_negotiation.cc:407] Connection header received
0502 14:09:57.976919 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.976954 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.977229 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.977445 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:57.979141 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.979909 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.981449 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.982942 (+ 1493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.983094 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.983275 (+ 181us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.983458 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20240502 14:09:57.985139 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.974473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40511 (local address 127.27.174.66:57455)
0502 14:09:57.976339 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.976380 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.976644 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.977890 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.977921 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.978014 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:57.978884 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.978919 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.980128 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.980147 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.981204 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.981236 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.981535 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.981615 (+ 80us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.981731 (+ 116us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.981936 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":1227,"threads_started":1}
I20240502 14:09:57.986218 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.976431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.67:42119 (local address 127.27.174.66:47561)
0502 14:09:57.978707 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:57.978747 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0502 14:09:57.978974 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:09:57.979576 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:09:57.979601 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:09:57.979687 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 14:09:57.980426 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.980458 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.981586 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.981605 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:09:57.983222 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:09:57.983275 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.983580 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.983664 (+ 84us) client_negotiation.cc:769] Sending connection context
0502 14:09:57.983788 (+ 124us) client_negotiation.cc:240] Negotiation successful
0502 14:09:57.983975 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1301,"threads_started":1}
I20240502 14:09:57.986641 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:57.976756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.174.66:47561 (local address 127.27.174.67:42119)
0502 14:09:57.977597 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:57.977625 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:57.978857 (+ 1232us) server_negotiation.cc:407] Connection header received
0502 14:09:57.979194 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:57.979214 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:57.979403 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:57.979513 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0502 14:09:57.980650 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.981364 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:57.983470 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:57.984764 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:57.984889 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:57.985050 (+ 161us) server_negotiation.cc:299] Negotiation successful
0502 14:09:57.985376 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20240502 14:09:57.986749 10995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83332578555d434e93af9bd0fb059fec" candidate_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" is_pre_election: true
I20240502 14:09:57.987578 10995 raft_consensus.cc:2454] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e83cc6b4d4f4ab08f64bd47f7399002 in term 0.
I20240502 14:09:57.987938 11261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83332578555d434e93af9bd0fb059fec" candidate_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9263e6cd5dee484ab8dfddc94a52a772" is_pre_election: true
I20240502 14:09:57.988636 11261 raft_consensus.cc:2454] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e83cc6b4d4f4ab08f64bd47f7399002 in term 0.
I20240502 14:09:57.988654 11062 leader_election.cc:302] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c75e0336c34ab68fdc9e7e005bda2e, 8e83cc6b4d4f4ab08f64bd47f7399002; no voters:
I20240502 14:09:57.989367 11333 raft_consensus.cc:2781] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:09:57.989590 11333 raft_consensus.cc:482] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:09:57.989821 11333 raft_consensus.cc:3037] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:57.993911 11333 raft_consensus.cc:504] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:57.995095 11333 leader_election.cc:288] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [CANDIDATE]: Term 1 election: Requested vote from peers 53c75e0336c34ab68fdc9e7e005bda2e (127.27.174.65:40511), 9263e6cd5dee484ab8dfddc94a52a772 (127.27.174.67:42119)
I20240502 14:09:57.995754 10995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83332578555d434e93af9bd0fb059fec" candidate_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c75e0336c34ab68fdc9e7e005bda2e"
I20240502 14:09:57.995905 11261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83332578555d434e93af9bd0fb059fec" candidate_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9263e6cd5dee484ab8dfddc94a52a772"
I20240502 14:09:57.996153 10995 raft_consensus.cc:3037] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:57.996322 11261 raft_consensus.cc:3037] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:09:58.000584 10995 raft_consensus.cc:2454] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e83cc6b4d4f4ab08f64bd47f7399002 in term 1.
I20240502 14:09:58.000730 11261 raft_consensus.cc:2454] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e83cc6b4d4f4ab08f64bd47f7399002 in term 1.
I20240502 14:09:58.001332 11062 leader_election.cc:302] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c75e0336c34ab68fdc9e7e005bda2e, 8e83cc6b4d4f4ab08f64bd47f7399002; no voters:
I20240502 14:09:58.001924 11333 raft_consensus.cc:2781] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:09:58.003360 11333 raft_consensus.cc:686] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 1 LEADER]: Becoming Leader. State: Replica: 8e83cc6b4d4f4ab08f64bd47f7399002, State: Running, Role: LEADER
I20240502 14:09:58.004089 11333 consensus_queue.cc:237] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } }
I20240502 14:09:58.014237 10851 catalog_manager.cc:5484] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e83cc6b4d4f4ab08f64bd47f7399002 (127.27.174.66). New cstate: current_term: 1 leader_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e83cc6b4d4f4ab08f64bd47f7399002" member_type: VOTER last_known_addr { host: "127.27.174.66" port: 45929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 } health_report { overall_health: UNKNOWN } } }
I20240502 14:09:58.037524 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:58.031606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.27.174.66:45929)
0502 14:09:58.032040 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0502 14:09:58.032057 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:09:58.032088 (+ 31us) server_negotiation.cc:407] Connection header received
0502 14:09:58.032411 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:09:58.032427 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:09:58.032631 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:09:58.032759 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:09:58.033378 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:58.034149 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:09:58.035234 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:09:58.035556 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:09:58.035710 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:09:58.036188 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:09:58.036333 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0502 14:09:58.036551 (+ 218us) server_negotiation.cc:299] Negotiation successful
0502 14:09:58.036658 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20240502 14:09:58.049176 11261 raft_consensus.cc:1261] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 1 FOLLOWER]: Refusing update from remote peer 8e83cc6b4d4f4ab08f64bd47f7399002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:58.049443 10995 raft_consensus.cc:1261] T 83332578555d434e93af9bd0fb059fec P 53c75e0336c34ab68fdc9e7e005bda2e [term 1 FOLLOWER]: Refusing update from remote peer 8e83cc6b4d4f4ab08f64bd47f7399002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:09:58.050647 11337 consensus_queue.cc:1035] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9263e6cd5dee484ab8dfddc94a52a772" member_type: VOTER last_known_addr { host: "127.27.174.67" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:09:58.051535 11333 consensus_queue.cc:1035] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c75e0336c34ab68fdc9e7e005bda2e" member_type: VOTER last_known_addr { host: "127.27.174.65" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:09:58.069304 11340 mvcc.cc:205] Tried to move back new op lower bound from 7023243255994957824 to 7023243255831654400. Current Snapshot: MvccSnapshot[applied={T|T < 7023243255994957824}]
I20240502 14:09:58.088078 11342 mvcc.cc:205] Tried to move back new op lower bound from 7023243255994957824 to 7023243255831654400. Current Snapshot: MvccSnapshot[applied={T|T < 7023243255994957824}]
I20240502 14:09:58.105392 10851 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45768:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:58.106015 10851 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45768:
table { table_name: "test-table" } modify_external_catalogs: true
I20240502 14:09:58.116760 10851 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e22b3101e315498f94affcc8bbbbf207: Sending DeleteTablet for 3 replicas of tablet 83332578555d434e93af9bd0fb059fec
I20240502 14:09:58.118595 11108 tablet_service.cc:1514] Processing DeleteTablet for tablet 83332578555d434e93af9bd0fb059fec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:58 UTC) from {username='slave'} at 127.0.0.1:36944
I20240502 14:09:58.119414 10975 tablet_service.cc:1514] Processing DeleteTablet for tablet 83332578555d434e93af9bd0fb059fec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:58 UTC) from {username='slave'} at 127.0.0.1:35100
I20240502 14:09:58.126876 11241 tablet_service.cc:1514] Processing DeleteTablet for tablet 83332578555d434e93af9bd0fb059fec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:09:58 UTC) from {username='slave'} at 127.0.0.1:51712
I20240502 14:09:58.129436 11352 tablet_replica.cc:330] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: stopping tablet replica
I20240502 14:09:58.130975 11352 raft_consensus.cc:2229] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:09:58.132449 11352 raft_consensus.cc:2258] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:58.132648 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10911
I20240502 14:09:58.138568 11350 tablet_replica.cc:330] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: stopping tablet replica
I20240502 14:09:58.139844 11352 ts_tablet_manager.cc:1875] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:58.140713 11350 raft_consensus.cc:2229] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:09:58.143167 11350 raft_consensus.cc:2258] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:09:58.150523 11350 ts_tablet_manager.cc:1875] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:09:58.164438 11350 ts_tablet_manager.cc:1888] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:58.164911 11350 log.cc:1196] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-1/wal/wals/83332578555d434e93af9bd0fb059fec
I20240502 14:09:58.165136 11352 ts_tablet_manager.cc:1888] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 14:09:58.165589 11352 log.cc:1196] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Deleting WAL directory at /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714658657003751-28345-0/minicluster-data/ts-2/wal/wals/83332578555d434e93af9bd0fb059fec
I20240502 14:09:58.166342 11352 ts_tablet_manager.cc:1909] T 83332578555d434e93af9bd0fb059fec P 9263e6cd5dee484ab8dfddc94a52a772: Deleting consensus metadata
I20240502 14:09:58.167066 11350 ts_tablet_manager.cc:1909] T 83332578555d434e93af9bd0fb059fec P 8e83cc6b4d4f4ab08f64bd47f7399002: Deleting consensus metadata
I20240502 14:09:58.169538 10836 catalog_manager.cc:4830] TS 9263e6cd5dee484ab8dfddc94a52a772 (127.27.174.67:42119): tablet 83332578555d434e93af9bd0fb059fec (table test-table [id=7636ddadd7db44b195436fa7c86d2740]) successfully deleted
I20240502 14:09:58.170140 10837 catalog_manager.cc:4830] TS 8e83cc6b4d4f4ab08f64bd47f7399002 (127.27.174.66:45929): tablet 83332578555d434e93af9bd0fb059fec (table test-table [id=7636ddadd7db44b195436fa7c86d2740]) successfully deleted
W20240502 14:09:58.170568 10836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.174.65:40511 (error 108)
I20240502 14:09:58.171617 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11044
I20240502 14:09:58.175652 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:09:58.174078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.65:40511 (local address 127.0.0.1:35112)
0502 14:09:58.174852 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0502 14:09:58.175099 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20240502 14:09:58.176147 10836 catalog_manager.cc:4573] TS 53c75e0336c34ab68fdc9e7e005bda2e (127.27.174.65:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83332578555d434e93af9bd0fb059fec: Network error: Client connection negotiation failed: client connection to 127.27.174.65:40511: connect: Connection refused (error 111)
I20240502 14:09:58.197135 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11177
I20240502 14:09:58.218183 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 10818
2024-05-02T14:09:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (7216 ms)
[----------] 6 tests from EncryptionPolicyTest (47737 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:09:58 dist-test-slave-dlb8 krb5kdc[11356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:09:58 dist-test-slave-dlb8 krb5kdc[11356](info): set up 2 sockets
May 02 14:09:58 dist-test-slave-dlb8 krb5kdc[11356](info): commencing operation
krb5kdc: starting...
W20240502 14:09:59.181084 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.909s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:09:59 dist-test-slave-dlb8 krb5kdc[11356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:09:59.380549 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:09:59.663139 11369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:09:59.663695 11369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:09:59.664319 11369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:09:59.692660 11369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:09:59.692979 11369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:09:59.693280 11369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:09:59.693499 11369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:09:59.693711 11369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:09:59.726509 11369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:09:59.727676 11369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:09:59.728713 11369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:09:59.744035 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:59.744626 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:09:59.745293 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:09:59.746196 11369 server_base.cc:1034] running on GCE node
I20240502 14:10:00.796602 11369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:00.799163 11369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:00.800501 11369 hybrid_clock.cc:648] HybridClock initialized: now 1714659000800434 us; error 96 us; skew 500 ppm
I20240502 14:10:00.801306 11369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:00 dist-test-slave-dlb8 krb5kdc[11356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:00.807032 11369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:00.811326 11369 webserver.cc:460] Webserver started at http://127.0.0.1:37599/ using document root <none> and password file <none>
I20240502 14:10:00.812147 11369 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:00.812323 11369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:00.812705 11369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:00.817644 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "da797cc1305f4626b2a3ef1d1c3c36c9"
format_stamp: "Formatted at 2024-05-02 14:10:00 on dist-test-slave-dlb8"
server_key: "b8a1eeaa2ae799756a8621b3c76a3005"
server_key_iv: "d3efdf9ff57ed707210e4c665a758dca"
server_key_version: "encryptionkey@0"
I20240502 14:10:00.818790 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "da797cc1305f4626b2a3ef1d1c3c36c9"
format_stamp: "Formatted at 2024-05-02 14:10:00 on dist-test-slave-dlb8"
server_key: "b8a1eeaa2ae799756a8621b3c76a3005"
server_key_iv: "d3efdf9ff57ed707210e4c665a758dca"
server_key_version: "encryptionkey@0"
I20240502 14:10:00.825611 11369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:10:00.830735 11386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:00.831728 11369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:10:00.832031 11369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "da797cc1305f4626b2a3ef1d1c3c36c9"
format_stamp: "Formatted at 2024-05-02 14:10:00 on dist-test-slave-dlb8"
server_key: "b8a1eeaa2ae799756a8621b3c76a3005"
server_key_iv: "d3efdf9ff57ed707210e4c665a758dca"
server_key_version: "encryptionkey@0"
I20240502 14:10:00.832367 11369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:00.891343 11369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:00.896237 11369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:00.896664 11369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:00.960027 11369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37501
I20240502 14:10:00.960119 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37501 every 8 connection(s)
I20240502 14:10:00.962687 11369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:00.964632 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11369
I20240502 14:10:00.965281 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:00.966754 28345 external_mini_cluster.cc:1428] Setting key 928bc48000cdb35f40ac0b99ed401a2f
I20240502 14:10:00.968614 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:10:00 dist-test-slave-dlb8 krb5kdc[11356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714658999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240502 14:10:00.992133 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9: Bootstrap starting.
I20240502 14:10:00.997455 11438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:00.999311 11438 log.cc:824] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:00.999914 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:00.968837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49923 (local address 127.0.0.1:37501)
0502 14:10:00.971441 (+ 2604us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:00.971478 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:00.971556 (+ 78us) server_negotiation.cc:407] Connection header received
0502 14:10:00.977588 (+ 6032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:00.977636 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:00.978120 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:00.978573 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:00.980993 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:00.983259 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:00.984737 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:00.985316 (+ 579us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:00.989892 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:00.989977 (+ 85us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:00.990003 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:00.990149 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:00.993783 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:00.995156 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:00.995176 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:00.995205 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:00.995377 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:00.996020 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:00.996036 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:00.996049 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:00.996553 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:00.996870 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:00.997592 (+ 722us) server_negotiation.cc:299] Negotiation successful
0502 14:10:00.997945 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923,"thread_start_us":1543,"threads_started":1}
I20240502 14:10:01.004180 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:01.004298 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9: No bootstrap required, opened a new log
I20240502 14:10:01.024142 11438 raft_consensus.cc:348] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } }
I20240502 14:10:01.025111 11438 raft_consensus.cc:374] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:01.025342 11438 raft_consensus.cc:729] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da797cc1305f4626b2a3ef1d1c3c36c9, State: Initialized, Role: FOLLOWER
I20240502 14:10:01.025996 11438 consensus_queue.cc:260] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } }
I20240502 14:10:01.026494 11438 raft_consensus.cc:388] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:01.026757 11438 raft_consensus.cc:482] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:01.027030 11438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:01.033195 11438 raft_consensus.cc:504] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } }
I20240502 14:10:01.034024 11438 leader_election.cc:302] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da797cc1305f4626b2a3ef1d1c3c36c9; no voters:
I20240502 14:10:01.036067 11438 leader_election.cc:288] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:01.036513 11448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:01.038949 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:01.016828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41399 (local address 127.0.0.1:37501)
0502 14:10:01.017523 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:01.017551 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:01.018511 (+ 960us) server_negotiation.cc:407] Connection header received
0502 14:10:01.018793 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:01.018821 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:01.019041 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:01.019224 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:01.021530 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:01.022907 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:01.024200 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:01.024732 (+ 532us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:01.027227 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:01.027328 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:01.027362 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:01.027480 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:01.032410 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:01.033472 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:01.033502 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:01.033525 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:01.033705 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:01.034374 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:01.034400 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:01.034422 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:01.034919 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:01.035329 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:01.036136 (+ 807us) server_negotiation.cc:299] Negotiation successful
0502 14:10:01.036489 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20240502 14:10:01.040131 11448 raft_consensus.cc:686] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [term 1 LEADER]: Becoming Leader. State: Replica: da797cc1305f4626b2a3ef1d1c3c36c9, State: Running, Role: LEADER
I20240502 14:10:01.040907 11448 consensus_queue.cc:237] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } }
I20240502 14:10:01.041218 11438 sys_catalog.cc:564] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:01.051635 11450 sys_catalog.cc:455] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da797cc1305f4626b2a3ef1d1c3c36c9. Latest consensus state: current_term: 1 leader_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } } }
I20240502 14:10:01.052383 11450 sys_catalog.cc:458] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:01.050884 11449 sys_catalog.cc:455] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da797cc1305f4626b2a3ef1d1c3c36c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37501 } } }
I20240502 14:10:01.054994 11449 sys_catalog.cc:458] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:01.071774 11456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:01.082373 11456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:01.098191 11456 catalog_manager.cc:1347] Generated new cluster ID: c5bc8dedb6164de693593b1d141cd223
I20240502 14:10:01.098594 11456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:01.110567 11456 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:01.112385 11456 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:01.129212 11456 catalog_manager.cc:5857] T 00000000000000000000000000000000 P da797cc1305f4626b2a3ef1d1c3c36c9: Generated new TSK 0
I20240502 14:10:01.130296 11456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:01.148883 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11369
2024-05-02T14:10:01Z chronyd exiting
W20240502 14:10:01.169503 11354 connection.cc:535] client connection to 127.0.0.1:37501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37501): Connection reset by peer (error 104)
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (3019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:01 dist-test-slave-dlb8 krb5kdc[11469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:01 dist-test-slave-dlb8 krb5kdc[11469](info): set up 2 sockets
May 02 14:10:01 dist-test-slave-dlb8 krb5kdc[11469](info): commencing operation
krb5kdc: starting...
W20240502 14:10:02.220930 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:02 dist-test-slave-dlb8 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:02.428681 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:02.719499 11482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:02.720096 11482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:02.720698 11482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:02.750138 11482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:02.750519 11482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:02.750798 11482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:02.751030 11482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:02.751263 11482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:02.784727 11482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:02.785885 11482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:02.787107 11482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:02.800772 11488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:02.801405 11489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:02.804033 11482 server_base.cc:1034] running on GCE node
W20240502 14:10:02.802111 11491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:03.838589 11482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:03.841074 11482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:03.842388 11482 hybrid_clock.cc:648] HybridClock initialized: now 1714659003842350 us; error 53 us; skew 500 ppm
I20240502 14:10:03.843127 11482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:03 dist-test-slave-dlb8 krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:03.848712 11482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:03.853595 11482 webserver.cc:460] Webserver started at http://127.0.0.1:44295/ using document root <none> and password file <none>
I20240502 14:10:03.854427 11482 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:03.854601 11482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:03.854955 11482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:03.859804 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "3b676043cc1b4f959a678d73fa027959"
format_stamp: "Formatted at 2024-05-02 14:10:03 on dist-test-slave-dlb8"
server_key: "15a5d7a9259d35b943c143283c0958e9"
server_key_iv: "83bcb4519b00e732ad6764beb68e1838"
server_key_version: "encryptionkey@0"
I20240502 14:10:03.860873 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "3b676043cc1b4f959a678d73fa027959"
format_stamp: "Formatted at 2024-05-02 14:10:03 on dist-test-slave-dlb8"
server_key: "15a5d7a9259d35b943c143283c0958e9"
server_key_iv: "83bcb4519b00e732ad6764beb68e1838"
server_key_version: "encryptionkey@0"
I20240502 14:10:03.867575 11482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20240502 14:10:03.872706 11499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:03.873620 11482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:10:03.873880 11482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "3b676043cc1b4f959a678d73fa027959"
format_stamp: "Formatted at 2024-05-02 14:10:03 on dist-test-slave-dlb8"
server_key: "15a5d7a9259d35b943c143283c0958e9"
server_key_iv: "83bcb4519b00e732ad6764beb68e1838"
server_key_version: "encryptionkey@0"
I20240502 14:10:03.874200 11482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:03.965256 11482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:03.970288 11482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:03.970680 11482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:04.034634 11482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35389
I20240502 14:10:04.034716 11550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35389 every 8 connection(s)
I20240502 14:10:04.037186 11482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:04.040462 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11482
I20240502 14:10:04.040997 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:04.042452 28345 external_mini_cluster.cc:1428] Setting key 3f8ffd830fb71f9369eb6902162372c3
I20240502 14:10:04.043290 11551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:04.063374 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:04.044835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49249 (local address 127.0.0.1:35389)
0502 14:10:04.047584 (+ 2749us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:04.047609 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:04.047656 (+ 47us) server_negotiation.cc:407] Connection header received
0502 14:10:04.052768 (+ 5112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:04.052803 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:04.053117 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:04.053412 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:04.054999 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:04.056336 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:04.058225 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:04.058600 (+ 375us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:04.059814 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:04.059915 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:04.059957 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:04.060698 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:04.060811 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:04.060870 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:04.061012 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:04.061415 (+ 403us) server_negotiation.cc:299] Negotiation successful
0502 14:10:04.061565 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2222,"thread_start_us":1784,"threads_started":1}
I20240502 14:10:04.067370 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:04.067464 11551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959: Bootstrap starting.
I20240502 14:10:04.073957 11551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:04.076014 11551 log.cc:824] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:04.081086 11551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959: No bootstrap required, opened a new log
I20240502 14:10:04.093784 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:04.083937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57837 (local address 127.0.0.1:35389)
0502 14:10:04.084795 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:04.084821 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:04.085690 (+ 869us) server_negotiation.cc:407] Connection header received
0502 14:10:04.086013 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:04.086029 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:04.086208 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:04.086336 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:04.087688 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:04.088474 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:04.090292 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:04.090682 (+ 390us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:04.091835 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:04.091901 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:04.091918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:04.091960 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:04.092038 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:04.092084 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:04.092204 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:04.092556 (+ 352us) server_negotiation.cc:299] Negotiation successful
0502 14:10:04.092637 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20240502 14:10:04.103221 11551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } }
I20240502 14:10:04.103896 11551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:04.104103 11551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b676043cc1b4f959a678d73fa027959, State: Initialized, Role: FOLLOWER
I20240502 14:10:04.104795 11551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } }
I20240502 14:10:04.105278 11551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:04.105520 11551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:04.105800 11551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:04.109611 11551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } }
I20240502 14:10:04.110316 11551 leader_election.cc:302] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b676043cc1b4f959a678d73fa027959; no voters:
I20240502 14:10:04.111941 11551 leader_election.cc:288] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:04.112298 11561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:04.114382 11561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [term 1 LEADER]: Becoming Leader. State: Replica: 3b676043cc1b4f959a678d73fa027959, State: Running, Role: LEADER
I20240502 14:10:04.115165 11561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } }
I20240502 14:10:04.116010 11551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:04.126600 11563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b676043cc1b4f959a678d73fa027959. Latest consensus state: current_term: 1 leader_uuid: "3b676043cc1b4f959a678d73fa027959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } } }
I20240502 14:10:04.127331 11563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:04.129688 11562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b676043cc1b4f959a678d73fa027959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b676043cc1b4f959a678d73fa027959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35389 } } }
I20240502 14:10:04.130661 11562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:04.137041 11568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:04.151207 11568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:04.171208 11568 catalog_manager.cc:1347] Generated new cluster ID: 4728a2fce710415ead24186b37523b67
I20240502 14:10:04.171434 11568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:04.189756 11568 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:04.191046 11568 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:04.205143 11568 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3b676043cc1b4f959a678d73fa027959: Generated new TSK 0
I20240502 14:10:04.205981 11568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:04.216933 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11482
W20240502 14:10:04.237525 11467 connection.cc:535] client connection to 127.0.0.1:35389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35389): Connection reset by peer (error 104)
2024-05-02T14:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:04 dist-test-slave-dlb8 krb5kdc[11582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:04 dist-test-slave-dlb8 krb5kdc[11582](info): set up 2 sockets
May 02 14:10:04 dist-test-slave-dlb8 krb5kdc[11582](info): commencing operation
krb5kdc: starting...
W20240502 14:10:05.200866 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:05 dist-test-slave-dlb8 krb5kdc[11582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:05.398293 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:05.685846 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:05.686435 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:05.687043 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:05.714896 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:05.715276 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:05.715566 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:05.715801 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:05.716027 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:05.750319 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:05.751502 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:05.752529 11595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:05.767436 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:05.767910 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:05.769109 11595 server_base.cc:1034] running on GCE node
W20240502 14:10:05.768281 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:06.791375 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:06.793728 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:06.795032 11595 hybrid_clock.cc:648] HybridClock initialized: now 1714659006795000 us; error 52 us; skew 500 ppm
I20240502 14:10:06.795802 11595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:06 dist-test-slave-dlb8 krb5kdc[11582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:06.805727 11595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:06.810647 11595 webserver.cc:460] Webserver started at http://127.0.0.1:39767/ using document root <none> and password file <none>
I20240502 14:10:06.811512 11595 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:06.811681 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:06.812090 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:06.816839 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "531a4b6357ee430baa6fdb2472cf91c7"
format_stamp: "Formatted at 2024-05-02 14:10:06 on dist-test-slave-dlb8"
server_key: "a9a5897576ded43f7ddd2512c855520e"
server_key_iv: "71fc63de73370c7349b9f25f6466e9a8"
server_key_version: "encryptionkey@0"
I20240502 14:10:06.817898 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "531a4b6357ee430baa6fdb2472cf91c7"
format_stamp: "Formatted at 2024-05-02 14:10:06 on dist-test-slave-dlb8"
server_key: "a9a5897576ded43f7ddd2512c855520e"
server_key_iv: "71fc63de73370c7349b9f25f6466e9a8"
server_key_version: "encryptionkey@0"
I20240502 14:10:06.825232 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 14:10:06.830325 11612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:06.831154 11595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:10:06.831454 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "531a4b6357ee430baa6fdb2472cf91c7"
format_stamp: "Formatted at 2024-05-02 14:10:06 on dist-test-slave-dlb8"
server_key: "a9a5897576ded43f7ddd2512c855520e"
server_key_iv: "71fc63de73370c7349b9f25f6466e9a8"
server_key_version: "encryptionkey@0"
I20240502 14:10:06.831776 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:06.867731 11595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:06.873255 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:06.873692 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:06.936479 11595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37967
I20240502 14:10:06.936576 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37967 every 8 connection(s)
I20240502 14:10:06.938952 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:06.942956 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11595
I20240502 14:10:06.943555 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:06.944918 28345 external_mini_cluster.cc:1428] Setting key 838fa35f5cf4fe1557f70f38e27f7824
I20240502 14:10:06.945056 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:06.958922 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:06.947316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39261 (local address 127.0.0.1:37967)
0502 14:10:06.950149 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:06.950176 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:06.950237 (+ 61us) server_negotiation.cc:407] Connection header received
0502 14:10:06.954449 (+ 4212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:06.954482 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:06.954733 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:06.955018 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:06.956603 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:06.956624 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:06.956648 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:06.956685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:06.956784 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:06.956842 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:06.957016 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:06.957271 (+ 255us) server_negotiation.cc:299] Negotiation successful
0502 14:10:06.957430 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2276,"thread_start_us":1828,"threads_started":1}
I20240502 14:10:06.963554 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:06.970850 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7: Bootstrap starting.
I20240502 14:10:06.977710 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:06.979710 11664 log.cc:824] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:06.985527 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7: No bootstrap required, opened a new log
I20240502 14:10:06.985585 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:06.979513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54003 (local address 127.0.0.1:37967)
0502 14:10:06.980264 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:06.980290 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:06.981189 (+ 899us) server_negotiation.cc:407] Connection header received
0502 14:10:06.981632 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:06.981670 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:06.981899 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:06.982063 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:06.983587 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:06.983613 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:06.983639 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:06.983693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:06.983787 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:06.983857 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:06.984061 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:06.984192 (+ 131us) server_negotiation.cc:299] Negotiation successful
0502 14:10:06.984312 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20240502 14:10:07.003993 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } }
I20240502 14:10:07.004616 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:07.004794 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531a4b6357ee430baa6fdb2472cf91c7, State: Initialized, Role: FOLLOWER
I20240502 14:10:07.005362 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } }
I20240502 14:10:07.005766 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:07.005982 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:07.006266 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:07.010370 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } }
I20240502 14:10:07.010951 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531a4b6357ee430baa6fdb2472cf91c7; no voters:
I20240502 14:10:07.012475 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:07.012862 11674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:07.015008 11674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [term 1 LEADER]: Becoming Leader. State: Replica: 531a4b6357ee430baa6fdb2472cf91c7, State: Running, Role: LEADER
I20240502 14:10:07.015913 11674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } }
I20240502 14:10:07.018954 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:07.025317 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531a4b6357ee430baa6fdb2472cf91c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } } }
I20240502 14:10:07.025533 11675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531a4b6357ee430baa6fdb2472cf91c7. Latest consensus state: current_term: 1 leader_uuid: "531a4b6357ee430baa6fdb2472cf91c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531a4b6357ee430baa6fdb2472cf91c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37967 } } }
I20240502 14:10:07.026322 11675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:07.027876 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:07.030647 11680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:07.042008 11680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:07.058909 11680 catalog_manager.cc:1347] Generated new cluster ID: ef02342579a44b6dbf006d0f08690a20
I20240502 14:10:07.059237 11680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:07.089095 11680 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:07.089715 11680 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:07.105052 11680 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 531a4b6357ee430baa6fdb2472cf91c7: Generated new TSK 0
I20240502 14:10:07.106119 11680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:07.127295 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11595
2024-05-02T14:10:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:07 dist-test-slave-dlb8 krb5kdc[11695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:07 dist-test-slave-dlb8 krb5kdc[11695](info): set up 2 sockets
May 02 14:10:07 dist-test-slave-dlb8 krb5kdc[11695](info): commencing operation
krb5kdc: starting...
W20240502 14:10:08.076958 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.903s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:08 dist-test-slave-dlb8 krb5kdc[11695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:08.273895 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:08.551622 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:08.552188 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:08.552793 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:08.583340 11709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:08.583673 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:08.583982 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:08.584229 11709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:08.584481 11709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:08.617993 11709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:08.619125 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:08.620178 11709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:08.636809 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:08.637269 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:08.637857 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:08.638686 11709 server_base.cc:1034] running on GCE node
I20240502 14:10:09.681299 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:09.684011 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:09.685343 11709 hybrid_clock.cc:648] HybridClock initialized: now 1714659009685300 us; error 56 us; skew 500 ppm
I20240502 14:10:09.686177 11709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:09 dist-test-slave-dlb8 krb5kdc[11695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:09.691723 11709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:09.695986 11709 webserver.cc:460] Webserver started at http://127.0.0.1:34403/ using document root <none> and password file <none>
I20240502 14:10:09.696853 11709 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:09.697060 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:09.697484 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:09.702564 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "6331eb7da2674df0b5ca8d1980c8e20d"
format_stamp: "Formatted at 2024-05-02 14:10:09 on dist-test-slave-dlb8"
server_key: "3766eb544307d0ab79f1dd872e2cb60b"
server_key_iv: "9b0712e99668ace2993f401eaf0d185d"
server_key_version: "encryptionkey@0"
I20240502 14:10:09.703651 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "6331eb7da2674df0b5ca8d1980c8e20d"
format_stamp: "Formatted at 2024-05-02 14:10:09 on dist-test-slave-dlb8"
server_key: "3766eb544307d0ab79f1dd872e2cb60b"
server_key_iv: "9b0712e99668ace2993f401eaf0d185d"
server_key_version: "encryptionkey@0"
I20240502 14:10:09.710309 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 14:10:09.715322 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:09.716295 11709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:10:09.716598 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "6331eb7da2674df0b5ca8d1980c8e20d"
format_stamp: "Formatted at 2024-05-02 14:10:09 on dist-test-slave-dlb8"
server_key: "3766eb544307d0ab79f1dd872e2cb60b"
server_key_iv: "9b0712e99668ace2993f401eaf0d185d"
server_key_version: "encryptionkey@0"
I20240502 14:10:09.716938 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:09.775744 11709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:09.780740 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:09.781178 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:09.845922 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43479
I20240502 14:10:09.846004 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43479 every 8 connection(s)
I20240502 14:10:09.848425 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:09.850013 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11709
I20240502 14:10:09.850524 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:09.851840 28345 external_mini_cluster.cc:1428] Setting key 1d4cc17e692dfa8153dbf7ad04069c21
I20240502 14:10:09.854993 11778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 02 14:10:09 dist-test-slave-dlb8 krb5kdc[11695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240502 14:10:09.880334 11778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d: Bootstrap starting.
I20240502 14:10:09.885079 11778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:09.886804 11778 log.cc:824] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:09.888176 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:09.854393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50373 (local address 127.0.0.1:43479)
0502 14:10:09.857172 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:09.857194 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:09.857240 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:10:09.863930 (+ 6690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:09.863967 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:09.864290 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:09.864602 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:09.868625 (+ 4023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:09.870150 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:09.872003 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:09.872465 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:09.878282 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:09.878375 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:09.878402 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:09.878494 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:09.882474 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:09.883509 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:09.883533 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:09.883561 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:09.883718 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:09.884205 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:09.884223 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:09.884235 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:09.884737 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:09.885139 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:09.885787 (+ 648us) server_negotiation.cc:299] Negotiation successful
0502 14:10:09.886356 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2271,"thread_start_us":1820,"threads_started":1}
I20240502 14:10:09.892196 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:09.892400 11778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d: No bootstrap required, opened a new log
I20240502 14:10:09.911399 11778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } }
I20240502 14:10:09.912003 11778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:09.912232 11778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331eb7da2674df0b5ca8d1980c8e20d, State: Initialized, Role: FOLLOWER
I20240502 14:10:09.912968 11778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } }
I20240502 14:10:09.913589 11778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:09.913897 11778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:09.914253 11778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:09.918746 11778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } }
I20240502 14:10:09.919629 11778 leader_election.cc:302] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6331eb7da2674df0b5ca8d1980c8e20d; no voters:
I20240502 14:10:09.921655 11778 leader_election.cc:288] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:09.921989 11788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:09.925132 11788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [term 1 LEADER]: Becoming Leader. State: Replica: 6331eb7da2674df0b5ca8d1980c8e20d, State: Running, Role: LEADER
I20240502 14:10:09.925881 11788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } }
I20240502 14:10:09.926378 11778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:09.929066 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:09.907908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45551 (local address 127.0.0.1:43479)
0502 14:10:09.908755 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:09.908778 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:09.909468 (+ 690us) server_negotiation.cc:407] Connection header received
0502 14:10:09.909789 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:09.909819 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:09.910086 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:09.910290 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:09.912640 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:09.913737 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:09.915079 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:09.915568 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:09.917583 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:09.917670 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:09.917703 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:09.917824 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:09.922722 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:09.924300 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:09.924328 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:09.924351 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:09.924524 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:09.925236 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:09.925282 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:09.925301 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:09.925814 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:09.926182 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:09.926876 (+ 694us) server_negotiation.cc:299] Negotiation successful
0502 14:10:09.927273 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20240502 14:10:09.936982 11789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } } }
I20240502 14:10:09.937794 11789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:09.940474 11790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6331eb7da2674df0b5ca8d1980c8e20d. Latest consensus state: current_term: 1 leader_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6331eb7da2674df0b5ca8d1980c8e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43479 } } }
I20240502 14:10:09.941120 11790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:09.943395 11795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:09.952150 11795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:09.967278 11795 catalog_manager.cc:1347] Generated new cluster ID: 860ac7b183094cd6af215a03cff46672
I20240502 14:10:09.967535 11795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:09.993947 11795 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:09.996042 11795 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:10.018755 11795 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6331eb7da2674df0b5ca8d1980c8e20d: Generated new TSK 0
I20240502 14:10:10.019860 11795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:10.043557 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11709
2024-05-02T14:10:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:10 dist-test-slave-dlb8 krb5kdc[11809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:10 dist-test-slave-dlb8 krb5kdc[11809](info): set up 2 sockets
May 02 14:10:10 dist-test-slave-dlb8 krb5kdc[11809](info): commencing operation
krb5kdc: starting...
W20240502 14:10:11.001031 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.911s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:11 dist-test-slave-dlb8 krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:11.202381 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:11.482379 11822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:11.482954 11822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:11.483564 11822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:11.511960 11822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:11.512320 11822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:11.512616 11822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:11.512849 11822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:11.513077 11822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:11.546373 11822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:11.547575 11822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:11.548583 11822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:11.565073 11828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:11.565615 11829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:11.566130 11831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:11.566601 11822 server_base.cc:1034] running on GCE node
I20240502 14:10:12.604890 11822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:12.607345 11822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:12.608697 11822 hybrid_clock.cc:648] HybridClock initialized: now 1714659012608658 us; error 56 us; skew 500 ppm
I20240502 14:10:12.609462 11822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:12 dist-test-slave-dlb8 krb5kdc[11809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:12.621248 11822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:12.626363 11822 webserver.cc:460] Webserver started at http://127.0.0.1:35091/ using document root <none> and password file <none>
I20240502 14:10:12.627230 11822 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:12.627439 11822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:12.627879 11822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:12.632848 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "52d5246401a24d8099458a3cb3270e13"
format_stamp: "Formatted at 2024-05-02 14:10:12 on dist-test-slave-dlb8"
server_key: "716cb74fe7971d0a88bd0289ecc1d68f"
server_key_iv: "cdc1106681f5a6694a6fa49a3a57f59c"
server_key_version: "encryptionkey@0"
I20240502 14:10:12.633934 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "52d5246401a24d8099458a3cb3270e13"
format_stamp: "Formatted at 2024-05-02 14:10:12 on dist-test-slave-dlb8"
server_key: "716cb74fe7971d0a88bd0289ecc1d68f"
server_key_iv: "cdc1106681f5a6694a6fa49a3a57f59c"
server_key_version: "encryptionkey@0"
I20240502 14:10:12.640944 11822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:10:12.646813 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:12.647819 11822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:10:12.648118 11822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "52d5246401a24d8099458a3cb3270e13"
format_stamp: "Formatted at 2024-05-02 14:10:12 on dist-test-slave-dlb8"
server_key: "716cb74fe7971d0a88bd0289ecc1d68f"
server_key_iv: "cdc1106681f5a6694a6fa49a3a57f59c"
server_key_version: "encryptionkey@0"
I20240502 14:10:12.648494 11822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:12.709228 11822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:12.714193 11822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:12.714635 11822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:12.781100 11822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39785
I20240502 14:10:12.781180 11890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39785 every 8 connection(s)
I20240502 14:10:12.783821 11822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:12.784796 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11822
I20240502 14:10:12.785221 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:12.786252 28345 external_mini_cluster.cc:1428] Setting key 5b469d65cdbd3720a29728a3c6ebfca5
I20240502 14:10:12.790683 11891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:12.806224 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:12.789173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58239 (local address 127.0.0.1:39785)
0502 14:10:12.791708 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:12.791732 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:12.791781 (+ 49us) server_negotiation.cc:407] Connection header received
0502 14:10:12.796153 (+ 4372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:12.796197 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:12.796577 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:12.796944 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:12.798409 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:12.799902 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:12.801388 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:12.802023 (+ 635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:12.802896 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:12.803004 (+ 108us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:12.803030 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:12.803751 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:12.803864 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:12.803927 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:12.804064 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:12.804373 (+ 309us) server_negotiation.cc:299] Negotiation successful
0502 14:10:12.804647 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2062,"thread_start_us":1690,"threads_started":1}
I20240502 14:10:12.810343 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:12.812297 11891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13: Bootstrap starting.
I20240502 14:10:12.816807 11891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:12.818413 11891 log.cc:824] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:12.824545 11891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13: No bootstrap required, opened a new log
I20240502 14:10:12.833639 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:12.822647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52263 (local address 127.0.0.1:39785)
0502 14:10:12.823508 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:12.823540 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:12.824367 (+ 827us) server_negotiation.cc:407] Connection header received
0502 14:10:12.824701 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:12.824722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:12.824899 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:12.825029 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:12.827415 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:12.828633 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:12.830400 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:12.830747 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:12.831809 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:12.831872 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:12.831892 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:12.831931 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:12.832026 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:12.832070 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:12.832169 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:12.832435 (+ 266us) server_negotiation.cc:299] Negotiation successful
0502 14:10:12.832565 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20240502 14:10:12.846442 11891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } }
I20240502 14:10:12.847059 11891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:12.847301 11891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d5246401a24d8099458a3cb3270e13, State: Initialized, Role: FOLLOWER
I20240502 14:10:12.847982 11891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } }
I20240502 14:10:12.848445 11891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:12.848682 11891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:12.848949 11891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:12.852792 11891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } }
I20240502 14:10:12.853400 11891 leader_election.cc:302] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d5246401a24d8099458a3cb3270e13; no voters:
I20240502 14:10:12.855027 11891 leader_election.cc:288] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:12.855365 11901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:12.857977 11901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [term 1 LEADER]: Becoming Leader. State: Replica: 52d5246401a24d8099458a3cb3270e13, State: Running, Role: LEADER
I20240502 14:10:12.858760 11901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } }
I20240502 14:10:12.859555 11891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:12.868505 11902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d5246401a24d8099458a3cb3270e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } } }
I20240502 14:10:12.868607 11903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d5246401a24d8099458a3cb3270e13. Latest consensus state: current_term: 1 leader_uuid: "52d5246401a24d8099458a3cb3270e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d5246401a24d8099458a3cb3270e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39785 } } }
I20240502 14:10:12.869457 11902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:12.869493 11903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:12.874694 11909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:12.887183 11909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:12.903323 11909 catalog_manager.cc:1347] Generated new cluster ID: 2f57b35918e64d74b655f3d1a3ce1bbd
I20240502 14:10:12.903574 11909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:12.924132 11909 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:12.926084 11909 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:12.939469 11909 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 52d5246401a24d8099458a3cb3270e13: Generated new TSK 0
I20240502 14:10:12.940510 11909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:12.954797 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11822
2024-05-02T14:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:13 dist-test-slave-dlb8 krb5kdc[11922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:13 dist-test-slave-dlb8 krb5kdc[11922](info): set up 2 sockets
May 02 14:10:13 dist-test-slave-dlb8 krb5kdc[11922](info): commencing operation
krb5kdc: starting...
W20240502 14:10:13.905186 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:14 dist-test-slave-dlb8 krb5kdc[11922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:14.104694 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:14.389859 11935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:14.390442 11935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:14.391031 11935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:14.418566 11935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:14.418867 11935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:14.419085 11935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:14.419271 11935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:14.419445 11935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:14.452246 11935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:14.453299 11935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:14.454214 11935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:14.468008 11941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:14.468730 11944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:14.469906 11935 server_base.cc:1034] running on GCE node
W20240502 14:10:14.469333 11942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:15.489714 11935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:15.492192 11935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:15.493516 11935 hybrid_clock.cc:648] HybridClock initialized: now 1714659015493478 us; error 45 us; skew 500 ppm
I20240502 14:10:15.494225 11935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:15 dist-test-slave-dlb8 krb5kdc[11922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:15.499365 11935 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:15.504156 11935 webserver.cc:460] Webserver started at http://127.0.0.1:34787/ using document root <none> and password file <none>
I20240502 14:10:15.504971 11935 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:15.505138 11935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:15.505559 11935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:15.510388 11935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "3f9c4f192a8640929c0d76ce6f58a0b3"
format_stamp: "Formatted at 2024-05-02 14:10:15 on dist-test-slave-dlb8"
server_key: "46ed21ff42cac12e6d297c2194db12e1"
server_key_iv: "c5316afa6c2ef040dcefc903b849042d"
server_key_version: "encryptionkey@0"
I20240502 14:10:15.511466 11935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "3f9c4f192a8640929c0d76ce6f58a0b3"
format_stamp: "Formatted at 2024-05-02 14:10:15 on dist-test-slave-dlb8"
server_key: "46ed21ff42cac12e6d297c2194db12e1"
server_key_iv: "c5316afa6c2ef040dcefc903b849042d"
server_key_version: "encryptionkey@0"
I20240502 14:10:15.517974 11935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:10:15.523165 11952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:15.524092 11935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:10:15.524418 11935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "3f9c4f192a8640929c0d76ce6f58a0b3"
format_stamp: "Formatted at 2024-05-02 14:10:15 on dist-test-slave-dlb8"
server_key: "46ed21ff42cac12e6d297c2194db12e1"
server_key_iv: "c5316afa6c2ef040dcefc903b849042d"
server_key_version: "encryptionkey@0"
I20240502 14:10:15.524773 11935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:15.563611 11935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:15.568959 11935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:15.569386 11935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:15.635846 11935 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35295
I20240502 14:10:15.635931 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35295 every 8 connection(s)
I20240502 14:10:15.638286 11935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:15.642766 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 11935
I20240502 14:10:15.643347 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:15.644186 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:15.644862 28345 external_mini_cluster.cc:1428] Setting key 6cc70bd568e0eb044703560bbef138cb
I20240502 14:10:15.659672 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:15.647369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47473 (local address 127.0.0.1:35295)
0502 14:10:15.650297 (+ 2928us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:15.650325 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:15.650376 (+ 51us) server_negotiation.cc:407] Connection header received
0502 14:10:15.654423 (+ 4047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:15.654464 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:15.654740 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:15.655056 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:15.657104 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:15.657126 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:15.657149 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:15.657187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:15.657293 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:15.657364 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:15.657696 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:15.657991 (+ 295us) server_negotiation.cc:299] Negotiation successful
0502 14:10:15.658217 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2316,"thread_start_us":1852,"threads_started":1}
I20240502 14:10:15.663964 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:15.668828 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3: Bootstrap starting.
I20240502 14:10:15.673822 12004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:15.675534 12004 log.cc:824] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:15.680974 12004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3: No bootstrap required, opened a new log
I20240502 14:10:15.682765 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:15.676651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58961 (local address 127.0.0.1:35295)
0502 14:10:15.677543 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:15.677578 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:15.678856 (+ 1278us) server_negotiation.cc:407] Connection header received
0502 14:10:15.679319 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:15.679337 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:15.679469 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:15.679601 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:15.681008 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:15.681029 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:15.681062 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:15.681112 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:15.681190 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:15.681265 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:15.681448 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:15.681684 (+ 236us) server_negotiation.cc:299] Negotiation successful
0502 14:10:15.681814 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20240502 14:10:15.699999 12004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } }
I20240502 14:10:15.700695 12004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:15.701007 12004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9c4f192a8640929c0d76ce6f58a0b3, State: Initialized, Role: FOLLOWER
I20240502 14:10:15.701831 12004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } }
I20240502 14:10:15.702347 12004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:15.702595 12004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:15.702858 12004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:15.706555 12004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } }
I20240502 14:10:15.707170 12004 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9c4f192a8640929c0d76ce6f58a0b3; no voters:
I20240502 14:10:15.708797 12004 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:15.709147 12014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:15.712210 12014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9c4f192a8640929c0d76ce6f58a0b3, State: Running, Role: LEADER
I20240502 14:10:15.713021 12014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } }
I20240502 14:10:15.713480 12004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:15.723438 12015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } } }
I20240502 14:10:15.723649 12016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9c4f192a8640929c0d76ce6f58a0b3. Latest consensus state: current_term: 1 leader_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c4f192a8640929c0d76ce6f58a0b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35295 } } }
I20240502 14:10:15.724220 12015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:15.724309 12016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:15.732035 12022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:15.743618 12022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:15.759569 12022 catalog_manager.cc:1347] Generated new cluster ID: d523695c9004433fbb7df50a1d900ebf
I20240502 14:10:15.759810 12022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:15.779822 12022 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:15.780318 12022 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:15.792711 12022 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3f9c4f192a8640929c0d76ce6f58a0b3: Generated new TSK 0
I20240502 14:10:15.793428 12022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:15.802531 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 11935
2024-05-02T14:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:15 dist-test-slave-dlb8 krb5kdc[12035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:15 dist-test-slave-dlb8 krb5kdc[12035](info): set up 2 sockets
May 02 14:10:15 dist-test-slave-dlb8 krb5kdc[12035](info): commencing operation
krb5kdc: starting...
W20240502 14:10:16.741206 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.894s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:16 dist-test-slave-dlb8 krb5kdc[12035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:16.938589 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:17.209295 12048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:17.209897 12048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:17.210556 12048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:17.238654 12048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:17.239042 12048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:17.239326 12048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:17.239567 12048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:17.239814 12048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:17.273005 12048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:17.274097 12048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:17.275040 12048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:17.288591 12055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:17.289225 12054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:17.290936 12048 server_base.cc:1034] running on GCE node
W20240502 14:10:17.290336 12057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:18.303143 12048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:18.305588 12048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:18.306923 12048 hybrid_clock.cc:648] HybridClock initialized: now 1714659018306874 us; error 65 us; skew 500 ppm
I20240502 14:10:18.307670 12048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:18 dist-test-slave-dlb8 krb5kdc[12035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:18.312999 12048 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:18.317790 12048 webserver.cc:460] Webserver started at http://127.0.0.1:37467/ using document root <none> and password file <none>
I20240502 14:10:18.318643 12048 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:18.318818 12048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:18.319176 12048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:18.324050 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "8a14460ad3464435872616cf87a64c06"
format_stamp: "Formatted at 2024-05-02 14:10:18 on dist-test-slave-dlb8"
server_key: "ccbf866ce61992bfb1264a7f5248049c"
server_key_iv: "eb0dd30eea4f5120d0fa3d8c9477596f"
server_key_version: "encryptionkey@0"
I20240502 14:10:18.325042 12048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8a14460ad3464435872616cf87a64c06"
format_stamp: "Formatted at 2024-05-02 14:10:18 on dist-test-slave-dlb8"
server_key: "ccbf866ce61992bfb1264a7f5248049c"
server_key_iv: "eb0dd30eea4f5120d0fa3d8c9477596f"
server_key_version: "encryptionkey@0"
I20240502 14:10:18.331466 12048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240502 14:10:18.336236 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:18.337168 12048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:10:18.337460 12048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "8a14460ad3464435872616cf87a64c06"
format_stamp: "Formatted at 2024-05-02 14:10:18 on dist-test-slave-dlb8"
server_key: "ccbf866ce61992bfb1264a7f5248049c"
server_key_iv: "eb0dd30eea4f5120d0fa3d8c9477596f"
server_key_version: "encryptionkey@0"
I20240502 14:10:18.337775 12048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:18.385447 12048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:18.390785 12048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:18.391211 12048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:18.456764 12048 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34185
I20240502 14:10:18.456841 12117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34185 every 8 connection(s)
I20240502 14:10:18.459349 12048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:18.464308 12118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:18.468736 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12048
I20240502 14:10:18.469352 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:18.470654 28345 external_mini_cluster.cc:1428] Setting key e695ac46cc33b8959b0c605578622eb6
I20240502 14:10:18.487941 12118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06: Bootstrap starting.
May 02 14:10:18 dist-test-slave-dlb8 krb5kdc[12035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240502 14:10:18.493495 12118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:18.495728 12118 log.cc:824] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:18.501387 12118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06: No bootstrap required, opened a new log
I20240502 14:10:18.505578 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:18.473108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:34147 (local address 127.0.0.1:34185)
0502 14:10:18.475849 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:18.475889 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:18.475954 (+ 65us) server_negotiation.cc:407] Connection header received
0502 14:10:18.480727 (+ 4773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:18.480785 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:18.481248 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:18.481695 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:18.484371 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:18.486536 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:18.487857 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:18.488412 (+ 555us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:18.492843 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:18.492962 (+ 119us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:18.493001 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:18.493136 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:18.498533 (+ 5397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:18.499804 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:18.499833 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:18.499872 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:18.500088 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:18.500720 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:18.500748 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:18.500768 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:18.501568 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:18.502071 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:18.502637 (+ 566us) server_negotiation.cc:299] Negotiation successful
0502 14:10:18.503104 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2019,"thread_start_us":1603,"threads_started":1}
I20240502 14:10:18.509876 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:18.521931 12118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } }
I20240502 14:10:18.522607 12118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:18.522835 12118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a14460ad3464435872616cf87a64c06, State: Initialized, Role: FOLLOWER
I20240502 14:10:18.523516 12118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } }
I20240502 14:10:18.524030 12118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:18.524295 12118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:18.524585 12118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:18.530663 12118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } }
I20240502 14:10:18.531615 12118 leader_election.cc:302] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a14460ad3464435872616cf87a64c06; no voters:
I20240502 14:10:18.534010 12118 leader_election.cc:288] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:18.534428 12128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:18.540339 12128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [term 1 LEADER]: Becoming Leader. State: Replica: 8a14460ad3464435872616cf87a64c06, State: Running, Role: LEADER
I20240502 14:10:18.541214 12118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:18.541218 12128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } }
I20240502 14:10:18.551702 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:18.525621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:37499 (local address 127.0.0.1:34185)
0502 14:10:18.526457 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:18.526488 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:18.527303 (+ 815us) server_negotiation.cc:407] Connection header received
0502 14:10:18.528728 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:18.528748 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:18.528931 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:18.529061 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:18.532479 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:18.533283 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:18.534673 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:18.535296 (+ 623us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:18.537754 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:18.537848 (+ 94us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:18.537887 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:18.538025 (+ 138us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:18.542349 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:18.543329 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:18.543358 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:18.543380 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:18.543529 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:18.544465 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:18.544513 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:18.544547 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:18.545100 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:18.547450 (+ 2350us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:18.547814 (+ 364us) server_negotiation.cc:299] Negotiation successful
0502 14:10:18.548221 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20240502 14:10:18.556298 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a14460ad3464435872616cf87a64c06. Latest consensus state: current_term: 1 leader_uuid: "8a14460ad3464435872616cf87a64c06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } } }
I20240502 14:10:18.557111 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:18.556545 12129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a14460ad3464435872616cf87a64c06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a14460ad3464435872616cf87a64c06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34185 } } }
I20240502 14:10:18.558751 12129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:18.562022 12138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:18.577641 12138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:18.595021 12138 catalog_manager.cc:1347] Generated new cluster ID: c2301705a4ad4c80b89ab42dcab124bf
I20240502 14:10:18.595337 12138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:18.615469 12138 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:18.616843 12138 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:18.630297 12138 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8a14460ad3464435872616cf87a64c06: Generated new TSK 0
I20240502 14:10:18.631163 12138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:18.653460 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12048
W20240502 14:10:18.675415 12033 connection.cc:535] client connection to 127.0.0.1:34185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34185): Connection reset by peer (error 104)
2024-05-02T14:10:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:18 dist-test-slave-dlb8 krb5kdc[12149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:18 dist-test-slave-dlb8 krb5kdc[12149](info): set up 2 sockets
May 02 14:10:18 dist-test-slave-dlb8 krb5kdc[12149](info): commencing operation
krb5kdc: starting...
W20240502 14:10:19.621413 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:19 dist-test-slave-dlb8 krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:19.827163 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:20.101464 12162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:20.102020 12162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:20.102556 12162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:20.129577 12162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:20.129904 12162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:20.130168 12162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:20.130380 12162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:20.130578 12162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:20.162876 12162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:20.163911 12162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:20.164814 12162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:20.179859 12168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:20.181023 12171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:20.181146 12162 server_base.cc:1034] running on GCE node
W20240502 14:10:20.180359 12169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:21.202416 12162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:21.204813 12162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:21.206149 12162 hybrid_clock.cc:648] HybridClock initialized: now 1714659021206117 us; error 50 us; skew 500 ppm
I20240502 14:10:21.206833 12162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:21 dist-test-slave-dlb8 krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:21.212417 12162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:21.216655 12162 webserver.cc:460] Webserver started at http://127.0.0.1:41697/ using document root <none> and password file <none>
I20240502 14:10:21.217442 12162 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:21.217638 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:21.218029 12162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:21.222903 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "f562fc7528804bf79755db8023008124"
format_stamp: "Formatted at 2024-05-02 14:10:21 on dist-test-slave-dlb8"
server_key: "af7e7835bb3514d37e564cbb6e78a0b5"
server_key_iv: "207bccf8c9dab77292a19f14266953dd"
server_key_version: "encryptionkey@0"
I20240502 14:10:21.223963 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "f562fc7528804bf79755db8023008124"
format_stamp: "Formatted at 2024-05-02 14:10:21 on dist-test-slave-dlb8"
server_key: "af7e7835bb3514d37e564cbb6e78a0b5"
server_key_iv: "207bccf8c9dab77292a19f14266953dd"
server_key_version: "encryptionkey@0"
I20240502 14:10:21.230574 12162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:10:21.235839 12179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:21.236749 12162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:10:21.237052 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "f562fc7528804bf79755db8023008124"
format_stamp: "Formatted at 2024-05-02 14:10:21 on dist-test-slave-dlb8"
server_key: "af7e7835bb3514d37e564cbb6e78a0b5"
server_key_iv: "207bccf8c9dab77292a19f14266953dd"
server_key_version: "encryptionkey@0"
I20240502 14:10:21.237406 12162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:21.281760 12162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:21.286867 12162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:21.287282 12162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:21.352221 12162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38847
I20240502 14:10:21.352290 12230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38847 every 8 connection(s)
I20240502 14:10:21.354676 12162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:21.357673 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12162
I20240502 14:10:21.358273 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:21.359692 28345 external_mini_cluster.cc:1428] Setting key 8554521f911f3ef9547c669144528a9f
I20240502 14:10:21.360934 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:21.381003 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:21.362277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:59073 (local address 127.0.0.1:38847)
0502 14:10:21.365310 (+ 3033us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:21.365336 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:21.365386 (+ 50us) server_negotiation.cc:407] Connection header received
0502 14:10:21.370537 (+ 5151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:21.370576 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:21.370938 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:21.371272 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:21.372906 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:21.374492 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:21.375931 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:21.376581 (+ 650us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:21.377449 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:21.377547 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:21.377582 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:21.378387 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:21.378502 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:21.378564 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:21.378713 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:21.379110 (+ 397us) server_negotiation.cc:299] Negotiation successful
0502 14:10:21.379288 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2501,"thread_start_us":2016,"threads_started":1}
I20240502 14:10:21.385493 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:21.385900 12231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124: Bootstrap starting.
I20240502 14:10:21.392721 12231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:21.394913 12231 log.cc:824] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:21.400367 12231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124: No bootstrap required, opened a new log
I20240502 14:10:21.413303 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:21.403097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:53055 (local address 127.0.0.1:38847)
0502 14:10:21.403981 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:21.404008 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:21.404841 (+ 833us) server_negotiation.cc:407] Connection header received
0502 14:10:21.405210 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:21.405228 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:21.405392 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:21.405517 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:21.407064 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:21.407954 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:21.409828 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:21.410223 (+ 395us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:21.411306 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:21.411376 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:21.411394 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:21.411438 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:21.411526 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:21.411579 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:21.411704 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:21.412038 (+ 334us) server_negotiation.cc:299] Negotiation successful
0502 14:10:21.412141 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20240502 14:10:21.424355 12231 raft_consensus.cc:348] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20240502 14:10:21.425056 12231 raft_consensus.cc:374] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:21.425285 12231 raft_consensus.cc:729] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f562fc7528804bf79755db8023008124, State: Initialized, Role: FOLLOWER
I20240502 14:10:21.425925 12231 consensus_queue.cc:260] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20240502 14:10:21.426488 12231 raft_consensus.cc:388] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:21.426744 12231 raft_consensus.cc:482] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:21.427029 12231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:21.431041 12231 raft_consensus.cc:504] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20240502 14:10:21.431910 12231 leader_election.cc:302] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f562fc7528804bf79755db8023008124; no voters:
I20240502 14:10:21.433705 12231 leader_election.cc:288] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:21.434067 12241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:21.436161 12241 raft_consensus.cc:686] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [term 1 LEADER]: Becoming Leader. State: Replica: f562fc7528804bf79755db8023008124, State: Running, Role: LEADER
I20240502 14:10:21.436934 12241 consensus_queue.cc:237] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20240502 14:10:21.437777 12231 sys_catalog.cc:564] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:21.450310 12243 sys_catalog.cc:455] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f562fc7528804bf79755db8023008124. Latest consensus state: current_term: 1 leader_uuid: "f562fc7528804bf79755db8023008124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20240502 14:10:21.451380 12243 sys_catalog.cc:458] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:21.453183 12242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f562fc7528804bf79755db8023008124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562fc7528804bf79755db8023008124" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20240502 14:10:21.453977 12242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:21.457679 12252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:21.466713 12252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:21.482445 12252 catalog_manager.cc:1347] Generated new cluster ID: a4bc3a4d9bb14e02b307658b3a292d75
I20240502 14:10:21.482764 12252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:21.498333 12252 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:21.500319 12252 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:21.519064 12252 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f562fc7528804bf79755db8023008124: Generated new TSK 0
I20240502 14:10:21.520133 12252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:21.537868 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12162
W20240502 14:10:21.559494 12147 connection.cc:535] client connection to 127.0.0.1:38847 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38847): Connection reset by peer (error 104)
2024-05-02T14:10:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:21 dist-test-slave-dlb8 krb5kdc[12262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:21 dist-test-slave-dlb8 krb5kdc[12262](info): set up 2 sockets
May 02 14:10:21 dist-test-slave-dlb8 krb5kdc[12262](info): commencing operation
krb5kdc: starting...
W20240502 14:10:22.493038 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.908s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:22 dist-test-slave-dlb8 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:22.697749 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:22.977947 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:22.978499 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:22.979028 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:23.006093 12275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:23.006444 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:23.006732 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:23.006958 12275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:23.007169 12275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:23.041105 12275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:23.042299 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:23.043242 12275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:23.056597 12281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:23.056960 12282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:23.059492 12284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:23.058254 12275 server_base.cc:1034] running on GCE node
I20240502 14:10:24.088008 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:24.090373 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:24.091693 12275 hybrid_clock.cc:648] HybridClock initialized: now 1714659024091661 us; error 51 us; skew 500 ppm
I20240502 14:10:24.092455 12275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:24 dist-test-slave-dlb8 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:24.102324 12275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:24.106566 12275 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
I20240502 14:10:24.107342 12275 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:24.107529 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:24.107913 12275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:24.112794 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "840ed5b434ce4c6599c4fa90dfd6266b"
format_stamp: "Formatted at 2024-05-02 14:10:24 on dist-test-slave-dlb8"
server_key: "5ad4c472f4ee4068a0e458a76d41ac8e"
server_key_iv: "e1f725587b33acce1c6937ed6894992a"
server_key_version: "encryptionkey@0"
I20240502 14:10:24.113831 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "840ed5b434ce4c6599c4fa90dfd6266b"
format_stamp: "Formatted at 2024-05-02 14:10:24 on dist-test-slave-dlb8"
server_key: "5ad4c472f4ee4068a0e458a76d41ac8e"
server_key_iv: "e1f725587b33acce1c6937ed6894992a"
server_key_version: "encryptionkey@0"
I20240502 14:10:24.120201 12275 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240502 14:10:24.125557 12292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:24.126483 12275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:10:24.126780 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "840ed5b434ce4c6599c4fa90dfd6266b"
format_stamp: "Formatted at 2024-05-02 14:10:24 on dist-test-slave-dlb8"
server_key: "5ad4c472f4ee4068a0e458a76d41ac8e"
server_key_iv: "e1f725587b33acce1c6937ed6894992a"
server_key_version: "encryptionkey@0"
I20240502 14:10:24.127102 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:24.164866 12275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:24.170892 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:24.171365 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:24.242484 12275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42415
I20240502 14:10:24.242607 12343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42415 every 8 connection(s)
I20240502 14:10:24.246127 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:24.248323 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12275
I20240502 14:10:24.248900 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:24.250367 28345 external_mini_cluster.cc:1428] Setting key 70feee58dec46a428ace728d476b86a4
I20240502 14:10:24.255407 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:24.270740 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:24.254362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:55239 (local address 127.0.0.1:42415)
0502 14:10:24.257531 (+ 3169us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:24.257566 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:24.257626 (+ 60us) server_negotiation.cc:407] Connection header received
0502 14:10:24.263789 (+ 6163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:24.263847 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:24.264266 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:24.264717 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:24.266687 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:24.266719 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:24.266762 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:24.266823 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:24.266966 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:24.267040 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:24.268108 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:24.268287 (+ 179us) server_negotiation.cc:299] Negotiation successful
0502 14:10:24.268518 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2490,"thread_start_us":1581,"threads_started":1}
I20240502 14:10:24.275911 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:24.282227 12344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b: Bootstrap starting.
I20240502 14:10:24.287073 12344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:24.289000 12344 log.cc:824] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:24.294621 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:24.288600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:46461 (local address 127.0.0.1:42415)
0502 14:10:24.289042 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:24.289056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:24.290194 (+ 1138us) server_negotiation.cc:407] Connection header received
0502 14:10:24.290488 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:24.290522 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:24.290762 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:24.290968 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:24.292514 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:24.292560 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:24.292575 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:24.292615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:24.292696 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:24.292743 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:24.292887 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:24.293165 (+ 278us) server_negotiation.cc:299] Negotiation successful
0502 14:10:24.293311 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20240502 14:10:24.295336 12344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b: No bootstrap required, opened a new log
I20240502 14:10:24.314121 12344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } }
I20240502 14:10:24.314749 12344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:24.314926 12344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840ed5b434ce4c6599c4fa90dfd6266b, State: Initialized, Role: FOLLOWER
I20240502 14:10:24.315533 12344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } }
I20240502 14:10:24.315943 12344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:24.316133 12344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:24.316377 12344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:24.320353 12344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } }
I20240502 14:10:24.320972 12344 leader_election.cc:302] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840ed5b434ce4c6599c4fa90dfd6266b; no voters:
I20240502 14:10:24.322650 12344 leader_election.cc:288] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:24.322993 12354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:24.326028 12354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [term 1 LEADER]: Becoming Leader. State: Replica: 840ed5b434ce4c6599c4fa90dfd6266b, State: Running, Role: LEADER
I20240502 14:10:24.326797 12354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } }
I20240502 14:10:24.327370 12344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:24.337047 12356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840ed5b434ce4c6599c4fa90dfd6266b. Latest consensus state: current_term: 1 leader_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } } }
I20240502 14:10:24.337922 12356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:24.338411 12355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ed5b434ce4c6599c4fa90dfd6266b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42415 } } }
I20240502 14:10:24.339028 12355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:24.342823 12364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:24.352934 12364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:24.368875 12364 catalog_manager.cc:1347] Generated new cluster ID: beb092b3e35043579144fb4d5c735160
I20240502 14:10:24.369160 12364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:24.392185 12364 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:24.392654 12364 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:24.405429 12364 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 840ed5b434ce4c6599c4fa90dfd6266b: Generated new TSK 0
I20240502 14:10:24.406486 12364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:24.424865 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12275
2024-05-02T14:10:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:24 dist-test-slave-dlb8 krb5kdc[12375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:24 dist-test-slave-dlb8 krb5kdc[12375](info): set up 2 sockets
May 02 14:10:24 dist-test-slave-dlb8 krb5kdc[12375](info): commencing operation
krb5kdc: starting...
W20240502 14:10:25.365101 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.898s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:25 dist-test-slave-dlb8 krb5kdc[12375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:25.569887 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:25.851545 12388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:25.852160 12388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:25.852742 12388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:25.880606 12388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:25.880985 12388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:25.881273 12388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:25.881513 12388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:25.881757 12388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:25.915354 12388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:25.916493 12388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:25.917510 12388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:25.931489 12394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:25.932374 12395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:25.934266 12388 server_base.cc:1034] running on GCE node
W20240502 14:10:25.933286 12397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:26.962433 12388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:26.965448 12388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:26.966859 12388 hybrid_clock.cc:648] HybridClock initialized: now 1714659026966834 us; error 43 us; skew 500 ppm
I20240502 14:10:26.967631 12388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:26 dist-test-slave-dlb8 krb5kdc[12375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:26.973263 12388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:26.978153 12388 webserver.cc:460] Webserver started at http://127.0.0.1:39247/ using document root <none> and password file <none>
I20240502 14:10:26.979043 12388 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:26.979234 12388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:26.979703 12388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:26.984637 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "8020064604d448038e1b33265ee5dbe4"
format_stamp: "Formatted at 2024-05-02 14:10:26 on dist-test-slave-dlb8"
server_key: "213b373119939edb87d45dc632d97d87"
server_key_iv: "b2094bc2396f7e9e2939931dea45a7bc"
server_key_version: "encryptionkey@0"
I20240502 14:10:26.985725 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8020064604d448038e1b33265ee5dbe4"
format_stamp: "Formatted at 2024-05-02 14:10:26 on dist-test-slave-dlb8"
server_key: "213b373119939edb87d45dc632d97d87"
server_key_iv: "b2094bc2396f7e9e2939931dea45a7bc"
server_key_version: "encryptionkey@0"
I20240502 14:10:26.992852 12388 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240502 14:10:26.997865 12405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:26.998787 12388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:10:26.999068 12388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "8020064604d448038e1b33265ee5dbe4"
format_stamp: "Formatted at 2024-05-02 14:10:26 on dist-test-slave-dlb8"
server_key: "213b373119939edb87d45dc632d97d87"
server_key_iv: "b2094bc2396f7e9e2939931dea45a7bc"
server_key_version: "encryptionkey@0"
I20240502 14:10:26.999451 12388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:27.051451 12388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:27.056491 12388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:27.056921 12388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:27.122337 12388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34927
I20240502 14:10:27.122411 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34927 every 8 connection(s)
I20240502 14:10:27.124823 12388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:27.129649 12457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:27.132153 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12388
I20240502 14:10:27.132623 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:27.133586 28345 external_mini_cluster.cc:1428] Setting key 0b111d1b33b9b4f1adfe77ec18f357ad
May 02 14:10:27 dist-test-slave-dlb8 krb5kdc[12375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240502 14:10:27.155017 12457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4: Bootstrap starting.
I20240502 14:10:27.161497 12457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:27.163743 12457 log.cc:824] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:27.167124 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:27.135502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:59131 (local address 127.0.0.1:34927)
0502 14:10:27.138403 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:27.138443 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:27.138522 (+ 79us) server_negotiation.cc:407] Connection header received
0502 14:10:27.142943 (+ 4421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:27.143002 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:27.143518 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:27.143975 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:27.146445 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:27.148564 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:27.149784 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:27.150376 (+ 592us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:27.154483 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:27.154613 (+ 130us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:27.154653 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:27.154795 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:27.160100 (+ 5305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:27.161306 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:27.161352 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:27.161403 (+ 51us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:27.161661 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:27.162362 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:27.162395 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:27.162417 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:27.163211 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:27.163690 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:27.164192 (+ 502us) server_negotiation.cc:299] Negotiation successful
0502 14:10:27.164645 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2113,"thread_start_us":1711,"threads_started":1}
I20240502 14:10:27.169098 12457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4: No bootstrap required, opened a new log
I20240502 14:10:27.171087 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:27.190205 12457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } }
I20240502 14:10:27.191068 12457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:27.191376 12457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8020064604d448038e1b33265ee5dbe4, State: Initialized, Role: FOLLOWER
I20240502 14:10:27.192281 12457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } }
I20240502 14:10:27.192979 12457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:27.193329 12457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:27.193720 12457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:27.198215 12457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } }
I20240502 14:10:27.198889 12457 leader_election.cc:302] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8020064604d448038e1b33265ee5dbe4; no voters:
I20240502 14:10:27.200897 12457 leader_election.cc:288] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:27.201251 12467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:27.203672 12467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [term 1 LEADER]: Becoming Leader. State: Replica: 8020064604d448038e1b33265ee5dbe4, State: Running, Role: LEADER
I20240502 14:10:27.204511 12467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } }
I20240502 14:10:27.206526 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:27.183646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:39675 (local address 127.0.0.1:34927)
0502 14:10:27.184445 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:27.184470 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:27.185215 (+ 745us) server_negotiation.cc:407] Connection header received
0502 14:10:27.185553 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:27.185569 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:27.185718 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:27.185839 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:27.189006 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:27.189820 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:27.191564 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:27.191975 (+ 411us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:27.195543 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:27.195607 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:27.195624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:27.195713 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:27.198933 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:27.200362 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:27.200404 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:27.200436 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:27.200624 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:27.201438 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:27.201476 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:27.201510 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:27.202150 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:27.202595 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:27.204640 (+ 2045us) server_negotiation.cc:299] Negotiation successful
0502 14:10:27.204991 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20240502 14:10:27.207562 12457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:27.216310 12469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8020064604d448038e1b33265ee5dbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } } }
I20240502 14:10:27.217020 12469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:27.217037 12468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8020064604d448038e1b33265ee5dbe4. Latest consensus state: current_term: 1 leader_uuid: "8020064604d448038e1b33265ee5dbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020064604d448038e1b33265ee5dbe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34927 } } }
I20240502 14:10:27.217720 12468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:27.221063 12474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:27.231333 12474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:27.247679 12474 catalog_manager.cc:1347] Generated new cluster ID: ceafc3feee1741f98f73dba7b7cca655
I20240502 14:10:27.248031 12474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:27.272563 12474 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:27.274572 12474 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:27.294319 12474 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8020064604d448038e1b33265ee5dbe4: Generated new TSK 0
I20240502 14:10:27.295331 12474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:27.318995 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12388
W20240502 14:10:27.340682 12373 connection.cc:535] client connection to 127.0.0.1:34927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34927): Connection reset by peer (error 104)
2024-05-02T14:10:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:27 dist-test-slave-dlb8 krb5kdc[12488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:27 dist-test-slave-dlb8 krb5kdc[12488](info): set up 2 sockets
May 02 14:10:27 dist-test-slave-dlb8 krb5kdc[12488](info): commencing operation
krb5kdc: starting...
W20240502 14:10:28.277403 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.912s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:28 dist-test-slave-dlb8 krb5kdc[12488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:28.473901 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:28.764250 12502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:28.764853 12502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:28.765415 12502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:28.794087 12502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:28.794520 12502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:28.794773 12502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:28.794965 12502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:28.795147 12502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:28.828055 12502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:28.829075 12502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:28.829982 12502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:28.843483 12508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:28.844244 12509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:28.845768 12502 server_base.cc:1034] running on GCE node
W20240502 14:10:28.844734 12511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:29.866649 12502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:29.869546 12502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:29.870934 12502 hybrid_clock.cc:648] HybridClock initialized: now 1714659029870898 us; error 56 us; skew 500 ppm
I20240502 14:10:29.871716 12502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:29 dist-test-slave-dlb8 krb5kdc[12488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:29.882615 12502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:29.887862 12502 webserver.cc:460] Webserver started at http://127.0.0.1:32801/ using document root <none> and password file <none>
I20240502 14:10:29.888693 12502 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:29.888890 12502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:29.889292 12502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:29.894600 12502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "ba0af1aaff63419a81bd121d3484d407"
format_stamp: "Formatted at 2024-05-02 14:10:29 on dist-test-slave-dlb8"
server_key: "ffd9fd476e6b2a174b6ba2ff4c1c48d4"
server_key_iv: "5d9e4508ae8bf527d628fa534cf03865"
server_key_version: "encryptionkey@0"
I20240502 14:10:29.895687 12502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "ba0af1aaff63419a81bd121d3484d407"
format_stamp: "Formatted at 2024-05-02 14:10:29 on dist-test-slave-dlb8"
server_key: "ffd9fd476e6b2a174b6ba2ff4c1c48d4"
server_key_iv: "5d9e4508ae8bf527d628fa534cf03865"
server_key_version: "encryptionkey@0"
I20240502 14:10:29.902164 12502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 14:10:29.907161 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:29.908048 12502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:10:29.908344 12502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "ba0af1aaff63419a81bd121d3484d407"
format_stamp: "Formatted at 2024-05-02 14:10:29 on dist-test-slave-dlb8"
server_key: "ffd9fd476e6b2a174b6ba2ff4c1c48d4"
server_key_iv: "5d9e4508ae8bf527d628fa534cf03865"
server_key_version: "encryptionkey@0"
I20240502 14:10:29.908674 12502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:29.975824 12502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:29.981081 12502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:29.981542 12502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:30.043819 12502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42267
I20240502 14:10:30.043916 12570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42267 every 8 connection(s)
I20240502 14:10:30.046231 12502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:30.049050 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12502
I20240502 14:10:30.049623 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:30.051136 28345 external_mini_cluster.cc:1428] Setting key d5f3d76d4441003d614188d5663662fe
I20240502 14:10:30.052492 12571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:30.072863 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:30.053734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:53609 (local address 127.0.0.1:42267)
0502 14:10:30.056546 (+ 2812us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:30.056570 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:30.056616 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:10:30.061770 (+ 5154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:30.061821 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:30.062225 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:30.062566 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:30.064518 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:30.065896 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:30.067891 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:30.068289 (+ 398us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:30.069415 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:30.069514 (+ 99us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:30.069542 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:30.070289 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:30.070395 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:30.070457 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:30.070600 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:30.070977 (+ 377us) server_negotiation.cc:299] Negotiation successful
0502 14:10:30.071137 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2296,"thread_start_us":1829,"threads_started":1}
I20240502 14:10:30.077289 12571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407: Bootstrap starting.
I20240502 14:10:30.077330 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:30.084537 12571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:30.086817 12571 log.cc:824] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:30.092731 12571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407: No bootstrap required, opened a new log
I20240502 14:10:30.107329 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:30.097140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:36169 (local address 127.0.0.1:42267)
0502 14:10:30.097815 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:30.097842 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:30.098839 (+ 997us) server_negotiation.cc:407] Connection header received
0502 14:10:30.099144 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:30.099171 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:30.099407 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:30.099600 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:30.100726 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:30.102112 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:30.103418 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:30.103952 (+ 534us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:30.104606 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:30.104698 (+ 92us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:30.104738 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:30.104809 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:30.104929 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:30.105002 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:30.105204 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:30.105469 (+ 265us) server_negotiation.cc:299] Negotiation successful
0502 14:10:30.105595 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20240502 14:10:30.116492 12571 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20240502 14:10:30.117156 12571 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:30.117383 12571 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0af1aaff63419a81bd121d3484d407, State: Initialized, Role: FOLLOWER
I20240502 14:10:30.118016 12571 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20240502 14:10:30.118520 12571 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:30.118760 12571 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:30.119072 12571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:30.123334 12571 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20240502 14:10:30.123994 12571 leader_election.cc:302] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0af1aaff63419a81bd121d3484d407; no voters:
I20240502 14:10:30.125774 12571 leader_election.cc:288] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:30.126252 12581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:30.128347 12581 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [term 1 LEADER]: Becoming Leader. State: Replica: ba0af1aaff63419a81bd121d3484d407, State: Running, Role: LEADER
I20240502 14:10:30.129372 12581 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20240502 14:10:30.131464 12571 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:30.140295 12583 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0af1aaff63419a81bd121d3484d407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } } }
I20240502 14:10:30.141235 12583 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:30.143289 12582 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0af1aaff63419a81bd121d3484d407. Latest consensus state: current_term: 1 leader_uuid: "ba0af1aaff63419a81bd121d3484d407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0af1aaff63419a81bd121d3484d407" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } } }
I20240502 14:10:30.143916 12582 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:30.146157 12589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:30.156237 12589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:30.171378 12589 catalog_manager.cc:1347] Generated new cluster ID: 0d3e1db69675461e824e7fea8af9660a
I20240502 14:10:30.171625 12589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:30.186093 12589 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:30.187268 12589 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:30.205914 12589 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ba0af1aaff63419a81bd121d3484d407: Generated new TSK 0
I20240502 14:10:30.206746 12589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:30.217291 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12502
W20240502 14:10:30.237644 12486 connection.cc:535] client connection to 127.0.0.1:42267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42267): Connection reset by peer (error 104)
2024-05-02T14:10:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:30 dist-test-slave-dlb8 krb5kdc[12602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:30 dist-test-slave-dlb8 krb5kdc[12602](info): set up 2 sockets
May 02 14:10:30 dist-test-slave-dlb8 krb5kdc[12602](info): commencing operation
krb5kdc: starting...
W20240502 14:10:31.188159 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:31 dist-test-slave-dlb8 krb5kdc[12602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240502 14:10:31.402319 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:31.683815 12615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:31.684363 12615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:31.684933 12615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:31.713246 12615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:31.713598 12615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:31.713867 12615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:31.714102 12615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:31.714318 12615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:31.748631 12615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:31.749830 12615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:31.750851 12615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:31.765244 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:31.765708 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:31.766472 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:31.767764 12615 server_base.cc:1034] running on GCE node
I20240502 14:10:32.798749 12615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:32.801280 12615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:32.802609 12615 hybrid_clock.cc:648] HybridClock initialized: now 1714659032802578 us; error 71 us; skew 500 ppm
I20240502 14:10:32.803414 12615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:32 dist-test-slave-dlb8 krb5kdc[12602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:32.808851 12615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240502 14:10:32.814330 12615 webserver.cc:460] Webserver started at http://127.0.0.1:40433/ using document root <none> and password file <none>
I20240502 14:10:32.815168 12615 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:32.815373 12615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:32.815807 12615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:32.820617 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "8c7bbf07f9e1453887f3af5242876cb0"
format_stamp: "Formatted at 2024-05-02 14:10:32 on dist-test-slave-dlb8"
server_key: "0e782f7dd3f64a64d07c5bcb7b3e1df3"
server_key_iv: "290414b5d43e7c2989b4f1dafde4aa5f"
server_key_version: "encryptionkey@0"
I20240502 14:10:32.821764 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "8c7bbf07f9e1453887f3af5242876cb0"
format_stamp: "Formatted at 2024-05-02 14:10:32 on dist-test-slave-dlb8"
server_key: "0e782f7dd3f64a64d07c5bcb7b3e1df3"
server_key_iv: "290414b5d43e7c2989b4f1dafde4aa5f"
server_key_version: "encryptionkey@0"
I20240502 14:10:32.830367 12615 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20240502 14:10:32.837881 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:32.838936 12615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:10:32.839251 12615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "8c7bbf07f9e1453887f3af5242876cb0"
format_stamp: "Formatted at 2024-05-02 14:10:32 on dist-test-slave-dlb8"
server_key: "0e782f7dd3f64a64d07c5bcb7b3e1df3"
server_key_iv: "290414b5d43e7c2989b4f1dafde4aa5f"
server_key_version: "encryptionkey@0"
I20240502 14:10:32.839612 12615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:32.876358 12615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:32.881851 12615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:32.882318 12615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:32.947252 12615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36193
I20240502 14:10:32.947335 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36193 every 8 connection(s)
I20240502 14:10:32.949702 12615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:32.954625 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12615
I20240502 14:10:32.954921 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:32.955047 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:32.956023 28345 external_mini_cluster.cc:1428] Setting key 24520557f9dc604efa5671e1511437d9
I20240502 14:10:32.972031 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:32.958159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:60167 (local address 127.0.0.1:36193)
0502 14:10:32.961097 (+ 2938us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:32.961123 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:32.961172 (+ 49us) server_negotiation.cc:407] Connection header received
0502 14:10:32.966971 (+ 5799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:32.967024 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:32.967377 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:32.967732 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:32.969448 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:32.969470 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:32.969497 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:32.969552 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:32.969681 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:32.969739 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:32.969916 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:32.970199 (+ 283us) server_negotiation.cc:299] Negotiation successful
0502 14:10:32.970432 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":1911,"threads_started":1}
I20240502 14:10:32.975925 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:32.979429 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0: Bootstrap starting.
I20240502 14:10:32.985865 12684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:32.988108 12684 log.cc:824] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:32.994547 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0: No bootstrap required, opened a new log
I20240502 14:10:32.998812 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:32.993391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:46831 (local address 127.0.0.1:36193)
0502 14:10:32.994238 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:32.994259 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:32.995159 (+ 900us) server_negotiation.cc:407] Connection header received
0502 14:10:32.995563 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:32.995579 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:32.995701 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:32.995816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:32.997191 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:32.997209 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:32.997224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:32.997262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:32.997339 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:32.997402 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:32.997540 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:32.997751 (+ 211us) server_negotiation.cc:299] Negotiation successful
0502 14:10:32.997836 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20240502 14:10:33.013358 12684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20240502 14:10:33.013976 12684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:33.014228 12684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7bbf07f9e1453887f3af5242876cb0, State: Initialized, Role: FOLLOWER
I20240502 14:10:33.014927 12684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20240502 14:10:33.015388 12684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:33.015622 12684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:33.015905 12684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:33.019512 12684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20240502 14:10:33.020128 12684 leader_election.cc:302] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c7bbf07f9e1453887f3af5242876cb0; no voters:
I20240502 14:10:33.021765 12684 leader_election.cc:288] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:33.022181 12694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:33.024840 12694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 8c7bbf07f9e1453887f3af5242876cb0, State: Running, Role: LEADER
I20240502 14:10:33.025579 12694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20240502 14:10:33.026424 12684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:33.035688 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7bbf07f9e1453887f3af5242876cb0. Latest consensus state: current_term: 1 leader_uuid: "8c7bbf07f9e1453887f3af5242876cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20240502 14:10:33.036324 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:33.037468 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c7bbf07f9e1453887f3af5242876cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7bbf07f9e1453887f3af5242876cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20240502 14:10:33.038244 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:33.039919 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:33.051335 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:33.066711 12704 catalog_manager.cc:1347] Generated new cluster ID: 69c48ad3c9a042f38e9178830e4d9749
I20240502 14:10:33.066954 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:33.086620 12704 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:33.087141 12704 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:33.098009 12704 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8c7bbf07f9e1453887f3af5242876cb0: Generated new TSK 0
I20240502 14:10:33.098798 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:33.109257 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12615
2024-05-02T14:10:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:33 dist-test-slave-dlb8 krb5kdc[12715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:33 dist-test-slave-dlb8 krb5kdc[12715](info): set up 2 sockets
May 02 14:10:33 dist-test-slave-dlb8 krb5kdc[12715](info): commencing operation
krb5kdc: starting...
W20240502 14:10:34.077062 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:34 dist-test-slave-dlb8 krb5kdc[12715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:34.282893 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34677
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:34.568459 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:34.569018 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:34.569636 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:34.597772 12728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:34.598135 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:34.598409 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:34.598627 12728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:34.598840 12728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:34.631882 12728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:34677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:34.633069 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:34.634093 12728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:34.648727 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:34.649422 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:34.649853 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:34.650550 12728 server_base.cc:1034] running on GCE node
I20240502 14:10:35.675211 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:35.677589 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:35.678951 12728 hybrid_clock.cc:648] HybridClock initialized: now 1714659035678898 us; error 70 us; skew 500 ppm
I20240502 14:10:35.679767 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:35 dist-test-slave-dlb8 krb5kdc[12715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:35.691193 12728 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:35.696218 12728 webserver.cc:460] Webserver started at http://127.27.174.126:44959/ using document root <none> and password file <none>
I20240502 14:10:35.697080 12728 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:35.697293 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:35.697716 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:35.702637 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "a0e54b889f9e4760a21c3d701fd6cf5e"
format_stamp: "Formatted at 2024-05-02 14:10:35 on dist-test-slave-dlb8"
server_key: "31fbc1130c4047741d8e4fb0def61cab"
server_key_iv: "b647039be39e8bb80c624c879460de7d"
server_key_version: "encryptionkey@0"
I20240502 14:10:35.703701 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "a0e54b889f9e4760a21c3d701fd6cf5e"
format_stamp: "Formatted at 2024-05-02 14:10:35 on dist-test-slave-dlb8"
server_key: "31fbc1130c4047741d8e4fb0def61cab"
server_key_iv: "b647039be39e8bb80c624c879460de7d"
server_key_version: "encryptionkey@0"
I20240502 14:10:35.710585 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:10:35.715754 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:35.716656 12728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:10:35.716969 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "a0e54b889f9e4760a21c3d701fd6cf5e"
format_stamp: "Formatted at 2024-05-02 14:10:35 on dist-test-slave-dlb8"
server_key: "31fbc1130c4047741d8e4fb0def61cab"
server_key_iv: "b647039be39e8bb80c624c879460de7d"
server_key_version: "encryptionkey@0"
I20240502 14:10:35.717322 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:35.775331 12728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:35.780107 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:35.780532 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:35.844978 12728 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:34677
I20240502 14:10:35.845055 12796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:34677 every 8 connection(s)
I20240502 14:10:35.847649 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:35.853032 12797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:35.854434 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12728
I20240502 14:10:35.854867 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:35.855834 28345 external_mini_cluster.cc:1428] Setting key 1bd1eb39266a6d5e37a4659af4dc3681
May 02 14:10:35 dist-test-slave-dlb8 krb5kdc[12715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:35.877794 12797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e: Bootstrap starting.
I20240502 14:10:35.884505 12797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:35.886724 12797 log.cc:824] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:35.890309 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:35.857735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42173 (local address 127.27.174.126:34677)
0502 14:10:35.860441 (+ 2706us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:35.860482 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:35.860562 (+ 80us) server_negotiation.cc:407] Connection header received
0502 14:10:35.865562 (+ 5000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:35.865618 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:35.866095 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:35.866536 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:35.868879 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:35.871038 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:35.872353 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:35.872957 (+ 604us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:35.877410 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:35.877533 (+ 123us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:35.877574 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:35.877710 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:35.883554 (+ 5844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:35.884713 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:35.884745 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:35.884787 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:35.885005 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:35.885636 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:35.885664 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:35.885688 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:35.886512 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:35.886960 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:35.887487 (+ 527us) server_negotiation.cc:299] Negotiation successful
0502 14:10:35.888007 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2002,"thread_start_us":1595,"threads_started":1}
I20240502 14:10:35.892331 12797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e: No bootstrap required, opened a new log
I20240502 14:10:35.894681 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:35.912335 12797 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } }
I20240502 14:10:35.913022 12797 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:35.913242 12797 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e54b889f9e4760a21c3d701fd6cf5e, State: Initialized, Role: FOLLOWER
I20240502 14:10:35.913888 12797 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } }
I20240502 14:10:35.914487 12797 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:35.914745 12797 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:35.915006 12797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:35.919363 12797 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } }
I20240502 14:10:35.919992 12797 leader_election.cc:302] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e54b889f9e4760a21c3d701fd6cf5e; no voters:
I20240502 14:10:35.922153 12797 leader_election.cc:288] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:35.922601 12807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:35.925002 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:35.906866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50287 (local address 127.27.174.126:34677)
0502 14:10:35.907346 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:35.907364 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:35.908601 (+ 1237us) server_negotiation.cc:407] Connection header received
0502 14:10:35.908934 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:35.908955 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:35.909100 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:35.909235 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:35.911459 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:35.912358 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:35.913706 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:35.914210 (+ 504us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:35.916613 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:35.916704 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:35.916725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:35.916812 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:35.920072 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:35.921065 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:35.921085 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:35.921099 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:35.921231 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:35.921806 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:35.921826 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:35.921839 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:35.922288 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:35.922655 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:35.923307 (+ 652us) server_negotiation.cc:299] Negotiation successful
0502 14:10:35.923634 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20240502 14:10:35.927569 12807 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [term 1 LEADER]: Becoming Leader. State: Replica: a0e54b889f9e4760a21c3d701fd6cf5e, State: Running, Role: LEADER
I20240502 14:10:35.928364 12807 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } }
I20240502 14:10:35.928920 12797 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:35.938078 12809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e54b889f9e4760a21c3d701fd6cf5e. Latest consensus state: current_term: 1 leader_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } } }
I20240502 14:10:35.938791 12809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:35.939621 12808 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e54b889f9e4760a21c3d701fd6cf5e" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 34677 } } }
I20240502 14:10:35.940248 12808 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:35.941774 12814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:35.953677 12814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:35.972273 12814 catalog_manager.cc:1347] Generated new cluster ID: e0fa86e4526f42b183a16d0b844c67c1
I20240502 14:10:35.972595 12814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:35.985862 12814 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:35.987299 12814 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:35.998927 12814 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a0e54b889f9e4760a21c3d701fd6cf5e: Generated new TSK 0
I20240502 14:10:35.999711 12814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:36.011694 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12728
W20240502 14:10:36.029111 12713 connection.cc:535] client connection to 127.27.174.126:34677 recv error: Network error: failed to read from TLS socket (remote: 127.27.174.126:34677): Connection reset by peer (error 104)
2024-05-02T14:10:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:36 dist-test-slave-dlb8 krb5kdc[12828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:36 dist-test-slave-dlb8 krb5kdc[12828](info): set up 2 sockets
May 02 14:10:36 dist-test-slave-dlb8 krb5kdc[12828](info): commencing operation
krb5kdc: starting...
W20240502 14:10:36.976975 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:37 dist-test-slave-dlb8 krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:37.180565 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41591
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:37.467252 12841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:37.467813 12841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:37.468453 12841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:37.496089 12841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:37.496500 12841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:37.496783 12841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:37.497025 12841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:37.497246 12841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:37.530377 12841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:37.531497 12841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:37.532562 12841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:37.547622 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:37.547870 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:37.548628 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:37.549391 12841 server_base.cc:1034] running on GCE node
I20240502 14:10:38.576337 12841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:38.579306 12841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:38.580722 12841 hybrid_clock.cc:648] HybridClock initialized: now 1714659038580691 us; error 58 us; skew 500 ppm
I20240502 14:10:38.581465 12841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:38 dist-test-slave-dlb8 krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:38.587158 12841 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:38.592447 12841 webserver.cc:460] Webserver started at http://127.27.174.126:35595/ using document root <none> and password file <none>
I20240502 14:10:38.593340 12841 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:38.593544 12841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:38.593967 12841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:38.598809 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "0e95a95974bd4448af92f966f80c9988"
format_stamp: "Formatted at 2024-05-02 14:10:38 on dist-test-slave-dlb8"
server_key: "0bff2d2fbcd57a43ccdfdac9e6e70af2"
server_key_iv: "91d8c5585c56d22e21324e2da07fe0fb"
server_key_version: "encryptionkey@0"
I20240502 14:10:38.599843 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "0e95a95974bd4448af92f966f80c9988"
format_stamp: "Formatted at 2024-05-02 14:10:38 on dist-test-slave-dlb8"
server_key: "0bff2d2fbcd57a43ccdfdac9e6e70af2"
server_key_iv: "91d8c5585c56d22e21324e2da07fe0fb"
server_key_version: "encryptionkey@0"
I20240502 14:10:38.606616 12841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20240502 14:10:38.611716 12859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:38.612696 12841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:10:38.613011 12841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "0e95a95974bd4448af92f966f80c9988"
format_stamp: "Formatted at 2024-05-02 14:10:38 on dist-test-slave-dlb8"
server_key: "0bff2d2fbcd57a43ccdfdac9e6e70af2"
server_key_iv: "91d8c5585c56d22e21324e2da07fe0fb"
server_key_version: "encryptionkey@0"
I20240502 14:10:38.613389 12841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:38.670539 12841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:38.676036 12841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:38.676492 12841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:38.746759 12841 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41591
I20240502 14:10:38.746840 12910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41591 every 8 connection(s)
I20240502 14:10:38.749500 12841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:38.754814 12911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:38.755930 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12841
I20240502 14:10:38.756775 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:38.758332 28345 external_mini_cluster.cc:1428] Setting key 21d5070596ff5069e6f5f0e3cccd20d8
I20240502 14:10:38.778115 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:38.760908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58097 (local address 127.27.174.126:41591)
0502 14:10:38.763730 (+ 2822us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:38.763757 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:38.763805 (+ 48us) server_negotiation.cc:407] Connection header received
0502 14:10:38.767691 (+ 3886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:38.767733 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:38.768144 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:38.768515 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:38.770156 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:38.771597 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:38.773132 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:38.773756 (+ 624us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:38.774701 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:38.774815 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:38.774849 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:38.775566 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:38.775676 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:38.775734 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:38.775906 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:38.776253 (+ 347us) server_negotiation.cc:299] Negotiation successful
0502 14:10:38.776448 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2157,"thread_start_us":1690,"threads_started":1}
I20240502 14:10:38.778555 12911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988: Bootstrap starting.
I20240502 14:10:38.782289 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:38.784096 12911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:38.786381 12911 log.cc:824] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:38.792208 12911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988: No bootstrap required, opened a new log
I20240502 14:10:38.811791 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:38.801946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38811 (local address 127.27.174.126:41591)
0502 14:10:38.802893 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:38.802918 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:38.803721 (+ 803us) server_negotiation.cc:407] Connection header received
0502 14:10:38.804056 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:38.804072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:38.804221 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:38.804352 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:38.805848 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:38.806704 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:38.808531 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:38.808855 (+ 324us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:38.809887 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:38.809948 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:38.809964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:38.810002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:38.810105 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:38.810154 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:38.810266 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:38.810592 (+ 326us) server_negotiation.cc:299] Negotiation successful
0502 14:10:38.810672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20240502 14:10:38.816932 12911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } }
I20240502 14:10:38.817551 12911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:38.817770 12911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e95a95974bd4448af92f966f80c9988, State: Initialized, Role: FOLLOWER
I20240502 14:10:38.818419 12911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } }
I20240502 14:10:38.818873 12911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:38.819164 12911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:38.819522 12911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:38.823549 12911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } }
I20240502 14:10:38.824321 12911 leader_election.cc:302] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e95a95974bd4448af92f966f80c9988; no voters:
I20240502 14:10:38.826009 12911 leader_election.cc:288] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:38.826408 12921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:38.829046 12921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [term 1 LEADER]: Becoming Leader. State: Replica: 0e95a95974bd4448af92f966f80c9988, State: Running, Role: LEADER
I20240502 14:10:38.829859 12921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } }
I20240502 14:10:38.831203 12911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:38.840335 12922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e95a95974bd4448af92f966f80c9988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } } }
I20240502 14:10:38.841182 12922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:38.842706 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e95a95974bd4448af92f966f80c9988. Latest consensus state: current_term: 1 leader_uuid: "0e95a95974bd4448af92f966f80c9988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e95a95974bd4448af92f966f80c9988" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41591 } } }
I20240502 14:10:38.843586 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:38.845952 12930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:38.857723 12930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:38.873950 12930 catalog_manager.cc:1347] Generated new cluster ID: bb0c7432f03f41d5855885619e2f2068
I20240502 14:10:38.874269 12930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:38.898483 12930 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:38.899856 12930 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:38.913631 12930 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0e95a95974bd4448af92f966f80c9988: Generated new TSK 0
I20240502 14:10:38.914426 12930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:38.929385 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12841
W20240502 14:10:38.950287 12826 connection.cc:535] client connection to 127.27.174.126:41591 recv error: Network error: failed to read from TLS socket (remote: 127.27.174.126:41591): Connection reset by peer (error 104)
2024-05-02T14:10:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:39 dist-test-slave-dlb8 krb5kdc[12942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:39 dist-test-slave-dlb8 krb5kdc[12942](info): set up 2 sockets
May 02 14:10:39 dist-test-slave-dlb8 krb5kdc[12942](info): commencing operation
krb5kdc: starting...
W20240502 14:10:39.872795 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.898s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:39 dist-test-slave-dlb8 krb5kdc[12942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:40.070544 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37827
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:40.348681 12955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:40.349277 12955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:40.349862 12955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:40.378499 12955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:40.378875 12955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:40.379168 12955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:40.379424 12955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:40.379647 12955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:40.414466 12955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:37827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:40.415566 12955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:40.416551 12955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:40.432432 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:40.432897 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:40.433427 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:40.434355 12955 server_base.cc:1034] running on GCE node
I20240502 14:10:41.480333 12955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:41.483418 12955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:41.484892 12955 hybrid_clock.cc:648] HybridClock initialized: now 1714659041484855 us; error 63 us; skew 500 ppm
I20240502 14:10:41.485649 12955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:41 dist-test-slave-dlb8 krb5kdc[12942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659041, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:41.495831 12955 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:41.501410 12955 webserver.cc:460] Webserver started at http://127.27.174.126:33963/ using document root <none> and password file <none>
I20240502 14:10:41.502386 12955 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:41.502610 12955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:41.503052 12955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:41.507964 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "783ec8e6879842cc89ccb6d90422a0f5"
format_stamp: "Formatted at 2024-05-02 14:10:41 on dist-test-slave-dlb8"
server_key: "050a490659b2b3f0c78262655d5faacf"
server_key_iv: "1a676cdf1c99c10e70baa0df199b99c4"
server_key_version: "encryptionkey@0"
I20240502 14:10:41.509060 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "783ec8e6879842cc89ccb6d90422a0f5"
format_stamp: "Formatted at 2024-05-02 14:10:41 on dist-test-slave-dlb8"
server_key: "050a490659b2b3f0c78262655d5faacf"
server_key_iv: "1a676cdf1c99c10e70baa0df199b99c4"
server_key_version: "encryptionkey@0"
I20240502 14:10:41.515969 12955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20240502 14:10:41.521584 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:41.522657 12955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 14:10:41.522976 12955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "783ec8e6879842cc89ccb6d90422a0f5"
format_stamp: "Formatted at 2024-05-02 14:10:41 on dist-test-slave-dlb8"
server_key: "050a490659b2b3f0c78262655d5faacf"
server_key_iv: "1a676cdf1c99c10e70baa0df199b99c4"
server_key_version: "encryptionkey@0"
I20240502 14:10:41.523342 12955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:41.562114 12955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:41.567713 12955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:41.568215 12955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:41.635917 12955 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:37827
I20240502 14:10:41.636004 13023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:37827 every 8 connection(s)
I20240502 14:10:41.638698 12955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:41.640393 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 12955
I20240502 14:10:41.641072 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:41.642800 28345 external_mini_cluster.cc:1428] Setting key 2f20632c739899daeda8484f777580e5
I20240502 14:10:41.645028 13024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:41.660068 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:41.645150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42699 (local address 127.27.174.126:37827)
0502 14:10:41.648313 (+ 3163us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:41.648346 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:41.648398 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:10:41.654981 (+ 6583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:41.655027 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:41.655346 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:41.655703 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:41.657538 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:41.657561 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:41.657588 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:41.657631 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:41.657731 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:41.657793 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:41.657995 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:41.658295 (+ 300us) server_negotiation.cc:299] Negotiation successful
0502 14:10:41.658463 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2600,"thread_start_us":2069,"threads_started":1}
I20240502 14:10:41.664462 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:41.669540 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5: Bootstrap starting.
I20240502 14:10:41.674736 13024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:41.677044 13024 log.cc:824] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:41.684336 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5: No bootstrap required, opened a new log
I20240502 14:10:41.685240 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:41.677808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56549 (local address 127.27.174.126:37827)
0502 14:10:41.678679 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:41.678728 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:41.680756 (+ 2028us) server_negotiation.cc:407] Connection header received
0502 14:10:41.681160 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:41.681180 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:41.681334 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:41.681465 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:41.682998 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:41.683027 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:41.683055 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:41.683110 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:41.683220 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:41.683308 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:41.683536 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:41.683742 (+ 206us) server_negotiation.cc:299] Negotiation successful
0502 14:10:41.683868 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20240502 14:10:41.703074 13024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } }
I20240502 14:10:41.703765 13024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:41.703967 13024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783ec8e6879842cc89ccb6d90422a0f5, State: Initialized, Role: FOLLOWER
I20240502 14:10:41.704665 13024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } }
I20240502 14:10:41.705251 13024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:41.705607 13024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:41.705878 13024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:41.709700 13024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } }
I20240502 14:10:41.710366 13024 leader_election.cc:302] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 783ec8e6879842cc89ccb6d90422a0f5; no voters:
I20240502 14:10:41.711959 13024 leader_election.cc:288] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:41.712335 13034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:41.714563 13034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [term 1 LEADER]: Becoming Leader. State: Replica: 783ec8e6879842cc89ccb6d90422a0f5, State: Running, Role: LEADER
I20240502 14:10:41.715936 13034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } }
I20240502 14:10:41.717365 13024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:41.726089 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "783ec8e6879842cc89ccb6d90422a0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } } }
I20240502 14:10:41.727026 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:41.729048 13035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783ec8e6879842cc89ccb6d90422a0f5. Latest consensus state: current_term: 1 leader_uuid: "783ec8e6879842cc89ccb6d90422a0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783ec8e6879842cc89ccb6d90422a0f5" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 37827 } } }
I20240502 14:10:41.730042 13035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:41.732468 13041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:41.743634 13041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:41.760536 13041 catalog_manager.cc:1347] Generated new cluster ID: fb139ddb69cd4a83a222c9d1451d4515
I20240502 14:10:41.760844 13041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:41.811688 13041 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:41.812188 13041 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:41.824460 13041 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 783ec8e6879842cc89ccb6d90422a0f5: Generated new TSK 0
I20240502 14:10:41.825457 13041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:41.840015 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 12955
2024-05-02T14:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:41 dist-test-slave-dlb8 krb5kdc[13055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:41 dist-test-slave-dlb8 krb5kdc[13055](info): set up 2 sockets
May 02 14:10:41 dist-test-slave-dlb8 krb5kdc[13055](info): commencing operation
krb5kdc: starting...
W20240502 14:10:42.800681 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:42 dist-test-slave-dlb8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:43.013722 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45473
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:43.296012 13068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:43.296523 13068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:43.297011 13068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:43.324847 13068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:43.325153 13068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:43.325382 13068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:43.325560 13068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:43.325731 13068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:43.358175 13068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:43.359243 13068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:43.360141 13068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:10:43.377122 13068 server_base.cc:1034] running on GCE node
W20240502 14:10:43.376165 13074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:43.376194 13077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:43.376885 13075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:44.434033 13068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:44.436542 13068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:44.437877 13068 hybrid_clock.cc:648] HybridClock initialized: now 1714659044437834 us; error 44 us; skew 500 ppm
I20240502 14:10:44.438719 13068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:44 dist-test-slave-dlb8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:44.444319 13068 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:44.448577 13068 webserver.cc:460] Webserver started at http://127.27.174.126:41405/ using document root <none> and password file <none>
I20240502 14:10:44.449393 13068 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:44.449597 13068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:44.450029 13068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:44.454994 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "77596e619d8f4fcdb70704faf8fc7b9f"
format_stamp: "Formatted at 2024-05-02 14:10:44 on dist-test-slave-dlb8"
server_key: "e19d321ec3a88b4824a784c3b180ba8d"
server_key_iv: "14d9287839f3563d55076fc7f04c23c5"
server_key_version: "encryptionkey@0"
I20240502 14:10:44.456036 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "77596e619d8f4fcdb70704faf8fc7b9f"
format_stamp: "Formatted at 2024-05-02 14:10:44 on dist-test-slave-dlb8"
server_key: "e19d321ec3a88b4824a784c3b180ba8d"
server_key_iv: "14d9287839f3563d55076fc7f04c23c5"
server_key_version: "encryptionkey@0"
I20240502 14:10:44.462810 13068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:10:44.468031 13085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:44.468936 13068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:10:44.469231 13068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "77596e619d8f4fcdb70704faf8fc7b9f"
format_stamp: "Formatted at 2024-05-02 14:10:44 on dist-test-slave-dlb8"
server_key: "e19d321ec3a88b4824a784c3b180ba8d"
server_key_iv: "14d9287839f3563d55076fc7f04c23c5"
server_key_version: "encryptionkey@0"
I20240502 14:10:44.469561 13068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:44.551090 13068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:44.556565 13068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:44.557008 13068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:44.623661 13068 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45473
I20240502 14:10:44.623754 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45473 every 8 connection(s)
I20240502 14:10:44.626310 13068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:44.631426 13137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:44.633102 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13068
I20240502 14:10:44.633561 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:44.634636 28345 external_mini_cluster.cc:1428] Setting key cbb71834e982a1620e8daee99baa90a7
May 02 14:10:44 dist-test-slave-dlb8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:44.657640 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f: Bootstrap starting.
I20240502 14:10:44.664646 13137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:44.666934 13137 log.cc:824] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:44.670452 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:44.636593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57683 (local address 127.27.174.126:45473)
0502 14:10:44.639347 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:44.639395 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:44.639466 (+ 71us) server_negotiation.cc:407] Connection header received
0502 14:10:44.644678 (+ 5212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:44.644740 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:44.645254 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:44.645760 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:44.648488 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:44.650626 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:44.651969 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:44.652591 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:44.657151 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:44.657297 (+ 146us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:44.657340 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:44.657485 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:44.663267 (+ 5782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:44.664377 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:44.664419 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:44.664475 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:44.664724 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:44.665295 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:44.665326 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:44.665351 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:44.666245 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:44.666720 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:44.667294 (+ 574us) server_negotiation.cc:299] Negotiation successful
0502 14:10:44.667878 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020,"thread_start_us":1614,"threads_started":1}
I20240502 14:10:44.672945 13137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f: No bootstrap required, opened a new log
I20240502 14:10:44.674896 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:44.697428 13137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } }
I20240502 14:10:44.698308 13137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:44.698549 13137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77596e619d8f4fcdb70704faf8fc7b9f, State: Initialized, Role: FOLLOWER
I20240502 14:10:44.699278 13137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } }
I20240502 14:10:44.699761 13137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:44.700003 13137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:44.700459 13137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:44.706423 13137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } }
I20240502 14:10:44.707289 13137 leader_election.cc:302] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77596e619d8f4fcdb70704faf8fc7b9f; no voters:
I20240502 14:10:44.709440 13137 leader_election.cc:288] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:44.709817 13147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:44.711934 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:44.690554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57783 (local address 127.27.174.126:45473)
0502 14:10:44.691422 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:44.691446 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:44.691491 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:10:44.691918 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:44.691944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:44.692157 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:44.692339 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:44.694839 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:44.695650 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:44.696938 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:44.697477 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:44.700154 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:44.700265 (+ 111us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:44.700301 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:44.700435 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:44.705167 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:44.706264 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:44.706300 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:44.706322 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:44.706514 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:44.707054 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:44.707080 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:44.707104 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:44.707647 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:44.707974 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:44.708549 (+ 575us) server_negotiation.cc:299] Negotiation successful
0502 14:10:44.709045 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20240502 14:10:44.712389 13147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [term 1 LEADER]: Becoming Leader. State: Replica: 77596e619d8f4fcdb70704faf8fc7b9f, State: Running, Role: LEADER
I20240502 14:10:44.713346 13147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } }
I20240502 14:10:44.715504 13137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:44.724087 13148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77596e619d8f4fcdb70704faf8fc7b9f. Latest consensus state: current_term: 1 leader_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } } }
I20240502 14:10:44.724845 13148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:44.724179 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77596e619d8f4fcdb70704faf8fc7b9f" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45473 } } }
I20240502 14:10:44.725687 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:44.728346 13154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:44.740741 13154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:44.760728 13154 catalog_manager.cc:1347] Generated new cluster ID: 0ba49576b04840468ab5525e428eaa98
I20240502 14:10:44.761040 13154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:44.761538 13164 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 14:10:44.777832 13154 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:44.779301 13154 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:44.792124 13154 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77596e619d8f4fcdb70704faf8fc7b9f: Generated new TSK 0
I20240502 14:10:44.792963 13154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:44.802399 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13068
2024-05-02T14:10:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:44 dist-test-slave-dlb8 krb5kdc[13168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:44 dist-test-slave-dlb8 krb5kdc[13168](info): set up 2 sockets
May 02 14:10:44 dist-test-slave-dlb8 krb5kdc[13168](info): commencing operation
krb5kdc: starting...
W20240502 14:10:45.788942 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:45 dist-test-slave-dlb8 krb5kdc[13168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:45.998399 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45711
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:46.282630 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:46.283275 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:46.283903 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:46.312719 13181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:46.313138 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:46.313441 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:46.313684 13181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:46.313915 13181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:46.348029 13181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:45711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:46.349149 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:46.350221 13181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:46.368744 13187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:46.369271 13190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:46.369720 13188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:46.370699 13181 server_base.cc:1034] running on GCE node
I20240502 14:10:47.422343 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:47.424897 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:47.426246 13181 hybrid_clock.cc:648] HybridClock initialized: now 1714659047426218 us; error 45 us; skew 500 ppm
I20240502 14:10:47.427073 13181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:47 dist-test-slave-dlb8 krb5kdc[13168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659047, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:47.437664 13181 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:47.442363 13181 webserver.cc:460] Webserver started at http://127.27.174.126:39667/ using document root <none> and password file <none>
I20240502 14:10:47.443305 13181 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:47.443529 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:47.443974 13181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:47.448967 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "088a4cb6fb684308b1d6352160d51544"
format_stamp: "Formatted at 2024-05-02 14:10:47 on dist-test-slave-dlb8"
server_key: "5ce1e792755bfade4612b1c6fd9d6330"
server_key_iv: "980ea2c286c64793d5cd50afbb859b47"
server_key_version: "encryptionkey@0"
I20240502 14:10:47.450145 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "088a4cb6fb684308b1d6352160d51544"
format_stamp: "Formatted at 2024-05-02 14:10:47 on dist-test-slave-dlb8"
server_key: "5ce1e792755bfade4612b1c6fd9d6330"
server_key_iv: "980ea2c286c64793d5cd50afbb859b47"
server_key_version: "encryptionkey@0"
I20240502 14:10:47.457180 13181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20240502 14:10:47.462450 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:47.463483 13181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:10:47.463882 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "088a4cb6fb684308b1d6352160d51544"
format_stamp: "Formatted at 2024-05-02 14:10:47 on dist-test-slave-dlb8"
server_key: "5ce1e792755bfade4612b1c6fd9d6330"
server_key_iv: "980ea2c286c64793d5cd50afbb859b47"
server_key_version: "encryptionkey@0"
I20240502 14:10:47.464387 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:47.520686 13181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:47.525970 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:47.526444 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:47.592221 13181 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:45711
I20240502 14:10:47.592308 13249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:45711 every 8 connection(s)
I20240502 14:10:47.594817 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:47.600937 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:47.605177 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13181
I20240502 14:10:47.605626 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:47.606688 28345 external_mini_cluster.cc:1428] Setting key 76cbcdb85f71d0f46c389becd7b7491a
I20240502 14:10:47.624295 13250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544: Bootstrap starting.
I20240502 14:10:47.627543 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:47.609025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33261 (local address 127.27.174.126:45711)
0502 14:10:47.612041 (+ 3016us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:47.612069 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:47.612121 (+ 52us) server_negotiation.cc:407] Connection header received
0502 14:10:47.616167 (+ 4046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:47.616203 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:47.616547 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:47.616862 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:47.618655 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:47.620118 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:47.622200 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:47.622630 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:47.623773 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:47.623868 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:47.623901 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:47.624726 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:47.624847 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:47.624905 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:47.625032 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:47.625453 (+ 421us) server_negotiation.cc:299] Negotiation successful
0502 14:10:47.625676 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2480,"thread_start_us":1978,"threads_started":1}
I20240502 14:10:47.630487 13250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:47.631942 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:47.632447 13250 log.cc:824] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:47.637984 13250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544: No bootstrap required, opened a new log
I20240502 14:10:47.656819 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:47.646625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57499 (local address 127.27.174.126:45711)
0502 14:10:47.647363 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:47.647386 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:47.648436 (+ 1050us) server_negotiation.cc:407] Connection header received
0502 14:10:47.648729 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:47.648758 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:47.649008 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:47.649199 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:47.650357 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:47.651566 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:47.652934 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:47.653498 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:47.654097 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:47.654195 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:47.654224 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:47.654303 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:47.654420 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:47.654497 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:47.654650 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:47.654875 (+ 225us) server_negotiation.cc:299] Negotiation successful
0502 14:10:47.655086 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20240502 14:10:47.662356 13250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } }
I20240502 14:10:47.663054 13250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:47.663270 13250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088a4cb6fb684308b1d6352160d51544, State: Initialized, Role: FOLLOWER
I20240502 14:10:47.663947 13250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } }
I20240502 14:10:47.664449 13250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:47.664676 13250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:47.664930 13250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:47.669170 13250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } }
I20240502 14:10:47.669821 13250 leader_election.cc:302] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088a4cb6fb684308b1d6352160d51544; no voters:
I20240502 14:10:47.671483 13250 leader_election.cc:288] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:47.671882 13260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:47.674958 13260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [term 1 LEADER]: Becoming Leader. State: Replica: 088a4cb6fb684308b1d6352160d51544, State: Running, Role: LEADER
I20240502 14:10:47.675863 13260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } }
I20240502 14:10:47.676251 13250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:47.687252 13262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088a4cb6fb684308b1d6352160d51544. Latest consensus state: current_term: 1 leader_uuid: "088a4cb6fb684308b1d6352160d51544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } } }
I20240502 14:10:47.687530 13261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088a4cb6fb684308b1d6352160d51544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088a4cb6fb684308b1d6352160d51544" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 45711 } } }
I20240502 14:10:47.688158 13262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:47.688171 13261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544 [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:47.691264 13269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:47.704085 13269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:47.721864 13269 catalog_manager.cc:1347] Generated new cluster ID: 91d7ad0d86de4e6ab3265ab13b3d28ea
I20240502 14:10:47.722314 13269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:47.736068 13269 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:47.737617 13269 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:47.751441 13269 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 088a4cb6fb684308b1d6352160d51544: Generated new TSK 0
I20240502 14:10:47.752342 13269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:47.762753 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13181
2024-05-02T14:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:47 dist-test-slave-dlb8 krb5kdc[13282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:47 dist-test-slave-dlb8 krb5kdc[13282](info): set up 2 sockets
May 02 14:10:47 dist-test-slave-dlb8 krb5kdc[13282](info): commencing operation
krb5kdc: starting...
W20240502 14:10:48.757349 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:48 dist-test-slave-dlb8 krb5kdc[13282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:48.968341 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46753
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:49.245860 13295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:49.246507 13295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:49.247063 13295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:49.274827 13295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:49.275215 13295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:49.275493 13295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:49.275722 13295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:49.275941 13295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:49.309157 13295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:49.310262 13295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:49.311195 13295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:49.324259 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:49.324318 13302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:50.398322 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:50.397449 13303 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1073 milliseconds
W20240502 14:10:50.400429 13295 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.079s user 0.358s sys 0.718s
W20240502 14:10:50.400681 13295 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.079s user 0.358s sys 0.718s
I20240502 14:10:50.400918 13295 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:10:50.402359 13295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:50.405025 13295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:50.406347 13295 hybrid_clock.cc:648] HybridClock initialized: now 1714659050406312 us; error 51 us; skew 500 ppm
I20240502 14:10:50.407124 13295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:50 dist-test-slave-dlb8 krb5kdc[13282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659050, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:50.413741 13295 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:50.418767 13295 webserver.cc:460] Webserver started at http://127.27.174.126:38355/ using document root <none> and password file <none>
I20240502 14:10:50.419625 13295 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:50.419802 13295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:50.420185 13295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:50.425069 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "a540ef1e439b4212bdf072318c4f59cc"
format_stamp: "Formatted at 2024-05-02 14:10:50 on dist-test-slave-dlb8"
server_key: "a486ef4651d7a5f8276e0de6ce2e57c4"
server_key_iv: "2a4abadb2c3375fecf7c2b714368dd09"
server_key_version: "encryptionkey@0"
I20240502 14:10:50.426210 13295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "a540ef1e439b4212bdf072318c4f59cc"
format_stamp: "Formatted at 2024-05-02 14:10:50 on dist-test-slave-dlb8"
server_key: "a486ef4651d7a5f8276e0de6ce2e57c4"
server_key_iv: "2a4abadb2c3375fecf7c2b714368dd09"
server_key_version: "encryptionkey@0"
I20240502 14:10:50.433912 13295 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20240502 14:10:50.439621 13312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:50.440732 13295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:10:50.441038 13295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "a540ef1e439b4212bdf072318c4f59cc"
format_stamp: "Formatted at 2024-05-02 14:10:50 on dist-test-slave-dlb8"
server_key: "a486ef4651d7a5f8276e0de6ce2e57c4"
server_key_iv: "2a4abadb2c3375fecf7c2b714368dd09"
server_key_version: "encryptionkey@0"
I20240502 14:10:50.441394 13295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:50.495584 13295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:50.501251 13295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:50.501672 13295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:50.567137 13295 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:46753
I20240502 14:10:50.567210 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:46753 every 8 connection(s)
I20240502 14:10:50.569684 13295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:50.572790 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13295
I20240502 14:10:50.573398 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:50.575002 28345 external_mini_cluster.cc:1428] Setting key 8eacc56c7bfd8fd20d4427cce4047dee
I20240502 14:10:50.576005 13364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:50.590538 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:50.577648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57361 (local address 127.27.174.126:46753)
0502 14:10:50.580668 (+ 3020us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:50.580693 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:50.580746 (+ 53us) server_negotiation.cc:407] Connection header received
0502 14:10:50.585618 (+ 4872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:50.585653 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:50.585931 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:50.586285 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:50.588075 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:50.588097 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:50.588121 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:50.588158 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:50.588266 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:50.588320 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:50.588587 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:50.588868 (+ 281us) server_negotiation.cc:299] Negotiation successful
0502 14:10:50.589080 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2480,"thread_start_us":1931,"threads_started":1}
I20240502 14:10:50.594918 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:50.599922 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc: Bootstrap starting.
I20240502 14:10:50.607679 13364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:50.609607 13364 log.cc:824] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:50.616259 13364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc: No bootstrap required, opened a new log
I20240502 14:10:50.620234 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:50.614748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51639 (local address 127.27.174.126:46753)
0502 14:10:50.615270 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:50.615288 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:50.616353 (+ 1065us) server_negotiation.cc:407] Connection header received
0502 14:10:50.616747 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:50.616768 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:50.616912 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:50.617070 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:50.618510 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:50.618530 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:50.618549 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0502 14:10:50.618588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:50.618671 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0502 14:10:50.618718 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:50.618902 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:50.619132 (+ 230us) server_negotiation.cc:299] Negotiation successful
0502 14:10:50.619276 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20240502 14:10:50.638224 13364 raft_consensus.cc:348] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } }
I20240502 14:10:50.638940 13364 raft_consensus.cc:374] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:10:50.639163 13364 raft_consensus.cc:729] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a540ef1e439b4212bdf072318c4f59cc, State: Initialized, Role: FOLLOWER
I20240502 14:10:50.640053 13364 consensus_queue.cc:260] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } }
I20240502 14:10:50.640667 13364 raft_consensus.cc:388] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:10:50.641001 13364 raft_consensus.cc:482] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:10:50.641325 13364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:50.645443 13364 raft_consensus.cc:504] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } }
I20240502 14:10:50.646224 13364 leader_election.cc:302] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a540ef1e439b4212bdf072318c4f59cc; no voters:
I20240502 14:10:50.647995 13364 leader_election.cc:288] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:10:50.648350 13374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:50.651437 13374 raft_consensus.cc:686] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [term 1 LEADER]: Becoming Leader. State: Replica: a540ef1e439b4212bdf072318c4f59cc, State: Running, Role: LEADER
I20240502 14:10:50.652212 13374 consensus_queue.cc:237] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } }
I20240502 14:10:50.653129 13364 sys_catalog.cc:564] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:50.658635 13375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a540ef1e439b4212bdf072318c4f59cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } } }
I20240502 14:10:50.659335 13375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:50.660493 13376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a540ef1e439b4212bdf072318c4f59cc. Latest consensus state: current_term: 1 leader_uuid: "a540ef1e439b4212bdf072318c4f59cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a540ef1e439b4212bdf072318c4f59cc" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 46753 } } }
I20240502 14:10:50.661031 13376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:50.668556 13381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:50.680023 13381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:50.701577 13381 catalog_manager.cc:1347] Generated new cluster ID: 8b7918ec5070415a91256a9ae38b5221
I20240502 14:10:50.701893 13381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:50.715796 13381 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:50.716348 13381 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:50.730253 13381 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a540ef1e439b4212bdf072318c4f59cc: Generated new TSK 0
I20240502 14:10:50.731079 13381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:50.744884 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13295
2024-05-02T14:10:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2982 ms)
[----------] 18 tests from AuthTokenIssuingTest (52520 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:50 dist-test-slave-dlb8 krb5kdc[13395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:50 dist-test-slave-dlb8 krb5kdc[13395](info): set up 2 sockets
May 02 14:10:50 dist-test-slave-dlb8 krb5kdc[13395](info): commencing operation
krb5kdc: starting...
W20240502 14:10:51.713259 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:51 dist-test-slave-dlb8 krb5kdc[13395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:51.928879 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42921
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:52.207232 13408 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:10:52.208052 13408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:52.208413 13408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:52.209007 13408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:52.238123 13408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:52.238528 13408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:52.238822 13408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:52.239064 13408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:52.239303 13408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:52.273769 13408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:52.274953 13408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:52.275961 13408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:52.291191 13414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:52.291177 13417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:52.291227 13415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:52.292517 13408 server_base.cc:1034] running on GCE node
I20240502 14:10:53.330927 13408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:53.333784 13408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:53.335203 13408 hybrid_clock.cc:648] HybridClock initialized: now 1714659053335156 us; error 81 us; skew 500 ppm
I20240502 14:10:53.335983 13408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:53 dist-test-slave-dlb8 krb5kdc[13395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:53.347160 13408 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:10:53.352246 13408 webserver.cc:460] Webserver started at http://127.27.174.126:37169/ using document root <none> and password file <none>
I20240502 14:10:53.353065 13408 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:53.353260 13408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:53.353710 13408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:53.358757 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "cff9edb101b04591bd0797953b7e672b"
format_stamp: "Formatted at 2024-05-02 14:10:53 on dist-test-slave-dlb8"
server_key: "2c131d29cf91bc03a4bb5b19e63223f0"
server_key_iv: "f03401b024f5da5609f1935642731559"
server_key_version: "encryptionkey@0"
I20240502 14:10:53.359865 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "cff9edb101b04591bd0797953b7e672b"
format_stamp: "Formatted at 2024-05-02 14:10:53 on dist-test-slave-dlb8"
server_key: "2c131d29cf91bc03a4bb5b19e63223f0"
server_key_iv: "f03401b024f5da5609f1935642731559"
server_key_version: "encryptionkey@0"
I20240502 14:10:53.366703 13408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 14:10:53.371846 13425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:53.372781 13408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:10:53.373121 13408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "cff9edb101b04591bd0797953b7e672b"
format_stamp: "Formatted at 2024-05-02 14:10:53 on dist-test-slave-dlb8"
server_key: "2c131d29cf91bc03a4bb5b19e63223f0"
server_key_iv: "f03401b024f5da5609f1935642731559"
server_key_version: "encryptionkey@0"
I20240502 14:10:53.373450 13408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:53.438151 13408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:53.443621 13408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:53.444067 13408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:53.511425 13408 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:42921
I20240502 14:10:53.511502 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:42921 every 8 connection(s)
I20240502 14:10:53.513947 13408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:10:53.519035 13477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:53.520083 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13408
I20240502 14:10:53.520500 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:10:53.521610 28345 external_mini_cluster.cc:1428] Setting key 06393703e5bb96298e917133cc1809da
I20240502 14:10:53.530684 13477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.125@KRBTEST.COM" created.
May 02 14:10:53 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:53.566807 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.536325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.27.174.126:42921)
0502 14:10:53.540434 (+ 4109us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:53.540474 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:53.540935 (+ 461us) server_negotiation.cc:407] Connection header received
0502 14:10:53.545865 (+ 4930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:53.545886 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:53.546194 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:53.546451 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:53.548589 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:53.550065 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:53.551322 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:53.551903 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:53.555692 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:53.555780 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:53.555803 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:53.555908 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:53.559671 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:53.560689 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:53.560710 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:53.560731 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:53.560889 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:53.561446 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:53.561465 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:53.561478 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:53.562117 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:53.562363 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:53.563178 (+ 815us) server_negotiation.cc:299] Negotiation successful
0502 14:10:53.563821 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":3254,"thread_start_us":2566,"threads_started":1}
I20240502 14:10:53.566869 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.536321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42921 (local address 127.0.0.1:34274)
0502 14:10:53.540680 (+ 4359us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.540737 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:53.545423 (+ 4686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:53.546519 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:53.546551 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:53.547532 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:53.548295 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:53.548340 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:53.550287 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:53.550307 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:53.551071 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:53.551117 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:53.551501 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:53.552511 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:53.552590 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:53.555464 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:53.559864 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:53.559897 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:53.559915 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:53.560501 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:53.561055 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:53.561072 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:53.561087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:53.561261 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:53.562328 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:53.562358 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:53.562638 (+ 280us) client_negotiation.cc:769] Sending connection context
0502 14:10:53.562942 (+ 304us) client_negotiation.cc:240] Negotiation successful
0502 14:10:53.563509 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3811,"spinlock_wait_cycles":18688,"thread_start_us":2827,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
I20240502 14:10:53.569895 13477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:10:53.570498 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:40715
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240502 14:10:53.572126 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.571064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40698)
0502 14:10:53.571571 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.571748 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20240502 14:10:53.572677 13426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:53.576897 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.575954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40714)
0502 14:10:53.576422 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.576556 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20240502 14:10:53.577252 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:53.626772 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 1
I20240502 14:10:53.629137 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.627985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40730)
0502 14:10:53.628553 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.628706 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20240502 14:10:53.631372 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.630361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40738)
0502 14:10:53.630868 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.631000 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:10:53.631774 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:53.700273 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 2
I20240502 14:10:53.702605 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.701407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40750)
0502 14:10:53.701985 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.702185 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20240502 14:10:53.704818 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.703714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40752)
0502 14:10:53.704231 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.704378 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20240502 14:10:53.705149 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:53.796607 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 3
I20240502 14:10:53.798949 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.797774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40756)
0502 14:10:53.798364 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.798508 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20240502 14:10:53.801119 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.800085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40770)
0502 14:10:53.800591 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.800730 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20240502 14:10:53.801463 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
W20240502 14:10:53.867365 13484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:10:53.868113 13484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:53.868393 13484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:53.868999 13484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:53.897926 13484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:53.898206 13484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:53.898476 13484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:53.898721 13484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:53.898932 13484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:53.932823 13484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:53.934002 13484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:53.935038 13484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:10:53.944921 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 4
I20240502 14:10:53.947909 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.946143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40772)
0502 14:10:53.947087 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.947343 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20240502 14:10:53.949352 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:53.951027 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:53.949772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40776)
0502 14:10:53.950490 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:53.950636 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
W20240502 14:10:53.951449 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:54.250893 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 5
I20240502 14:10:54.254146 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:54.251881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40788)
0502 14:10:54.252904 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:54.253206 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20240502 14:10:54.257380 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:54.255429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40796)
0502 14:10:54.256307 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:54.256593 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20240502 14:10:54.258078 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
I20240502 14:10:54.805648 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 6
I20240502 14:10:54.809762 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:54.807137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40812)
0502 14:10:54.809095 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:54.809247 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":1314,"threads_started":1}
I20240502 14:10:54.813216 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:54.812265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40824)
0502 14:10:54.812735 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:54.812869 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20240502 14:10:54.813591 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:40715: Network error: Client connection negotiation failed: client connection to 127.27.174.125:40715: connect: Connection refused (error 111)
W20240502 14:10:53.949987 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:53.954466 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:55.024114 13492 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240502 14:10:55.024240 13484 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240502 14:10:55.028358 13484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:55.030895 13484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:55.032238 13484 hybrid_clock.cc:648] HybridClock initialized: now 1714659055032210 us; error 45 us; skew 500 ppm
I20240502 14:10:55.033004 13484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:55 dist-test-slave-dlb8 krb5kdc[13395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:55.038828 13484 init.cc:377] Logged in from keytab as kudu/127.27.174.125@KRBTEST.COM (short username kudu)
I20240502 14:10:55.043713 13484 webserver.cc:460] Webserver started at http://127.27.174.125:40313/ using document root <none> and password file <none>
I20240502 14:10:55.044593 13484 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:55.044780 13484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:55.045181 13484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:55.050191 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data/instance:
uuid: "d5dee1ab184d44c99f4468b974a6def5"
format_stamp: "Formatted at 2024-05-02 14:10:55 on dist-test-slave-dlb8"
server_key: "39298642c4903631c65297c59296d821"
server_key_iv: "94988ca54beda74a86094c8cee19b25e"
server_key_version: "encryptionkey@0"
I20240502 14:10:55.051219 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal/instance:
uuid: "d5dee1ab184d44c99f4468b974a6def5"
format_stamp: "Formatted at 2024-05-02 14:10:55 on dist-test-slave-dlb8"
server_key: "39298642c4903631c65297c59296d821"
server_key_iv: "94988ca54beda74a86094c8cee19b25e"
server_key_version: "encryptionkey@0"
I20240502 14:10:55.058152 13484 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240502 14:10:55.063428 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:55.064463 13484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20240502 14:10:55.064791 13484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal
uuid: "d5dee1ab184d44c99f4468b974a6def5"
format_stamp: "Formatted at 2024-05-02 14:10:55 on dist-test-slave-dlb8"
server_key: "39298642c4903631c65297c59296d821"
server_key_iv: "94988ca54beda74a86094c8cee19b25e"
server_key_version: "encryptionkey@0"
I20240502 14:10:55.065151 13484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:55.109987 13484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:55.115183 13484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:55.115631 13484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:55.183194 13484 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:40715
I20240502 14:10:55.183271 13554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:40715 every 8 connection(s)
I20240502 14:10:55.186031 13484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
I20240502 14:10:55.190712 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:55.192843 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13484
I20240502 14:10:55.193248 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-1/wal/instance
I20240502 14:10:55.194326 28345 external_mini_cluster.cc:1428] Setting key 1303ac68eeba1c1bec78bdefb8bcf20b
I20240502 14:10:55.201594 13555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.124@KRBTEST.COM" created.
May 02 14:10:55 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:55.235564 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.206914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.27.174.126:42921)
0502 14:10:55.209157 (+ 2243us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:55.209185 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:55.210305 (+ 1120us) server_negotiation.cc:407] Connection header received
0502 14:10:55.214708 (+ 4403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:55.214733 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:55.214904 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:55.215068 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:55.217608 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.218606 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.220434 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.220788 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.225709 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:55.225798 (+ 89us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:55.225817 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:55.225906 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:55.229369 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.230883 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.230905 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.230919 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.231049 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.231824 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.231853 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.231881 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.232306 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:55.232482 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:55.233818 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0502 14:10:55.234174 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":1094,"threads_started":1}
I20240502 14:10:55.237093 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.207073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42921 (local address 127.0.0.1:34288)
0502 14:10:55.210001 (+ 2928us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.210103 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:55.214349 (+ 4246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:55.215282 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:55.215305 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:55.216067 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:55.217248 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.217321 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.218877 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.218913 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:55.220146 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.220203 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.220621 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.221606 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:55.221712 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:55.225413 (+ 3701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:55.229583 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.229622 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.229664 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.230701 (+ 1037us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.231222 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.231276 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.231307 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.231616 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.232533 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:55.232579 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:55.233170 (+ 591us) client_negotiation.cc:769] Sending connection context
0502 14:10:55.233665 (+ 495us) client_negotiation.cc:240] Negotiation successful
0502 14:10:55.234329 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":1864,"threads_started":1}
I20240502 14:10:55.239159 13555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
I20240502 14:10:55.246935 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:34141
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:10:55 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:10:55.262682 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.240475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40834)
0502 14:10:55.241380 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.241431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:55.241676 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:55.244075 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:55.244095 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:55.244717 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:55.245242 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.245293 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.247097 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.247126 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:55.248096 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.248154 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.248521 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.249346 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:55.249441 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:55.252695 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:55.257132 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.257151 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.257165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.257674 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.258277 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.258295 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.258310 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.258463 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.259293 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:55.259320 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:55.259559 (+ 239us) client_negotiation.cc:769] Sending connection context
0502 14:10:55.259785 (+ 226us) client_negotiation.cc:240] Negotiation successful
0502 14:10:55.260257 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20240502 14:10:55.263012 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.240839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.27.174.125:40715)
0502 14:10:55.243396 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:55.243419 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:55.243463 (+ 44us) server_negotiation.cc:407] Connection header received
0502 14:10:55.243559 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:55.243576 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:55.243806 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:55.243955 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:55.245519 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.246885 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.248363 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.248730 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.252942 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:55.253029 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:55.253058 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:55.253145 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:55.256935 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.257863 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.257886 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.257909 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.258100 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.258632 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.258653 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.258666 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.259097 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:55.259364 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:55.260052 (+ 688us) server_negotiation.cc:299] Negotiation successful
0502 14:10:55.260507 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2120,"thread_start_us":1386,"threads_started":1}
I20240502 14:10:55.265781 13555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:10:55.267967 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.266947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44492)
0502 14:10:55.267424 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.267603 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20240502 14:10:55.268494 13507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.272353 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.271472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44500)
0502 14:10:55.271893 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.272015 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20240502 14:10:55.272697 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.322196 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 1
I20240502 14:10:55.324996 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.323441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44516)
0502 14:10:55.324187 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.324407 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20240502 14:10:55.327879 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.326296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44528)
0502 14:10:55.327047 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.327285 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20240502 14:10:55.328338 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.396811 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 2
I20240502 14:10:55.399466 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.397828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44532)
0502 14:10:55.398598 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.398811 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20240502 14:10:55.402180 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.400660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44540)
0502 14:10:55.401348 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.401566 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20240502 14:10:55.402591 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.493996 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 3
I20240502 14:10:55.496564 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.495076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44556)
0502 14:10:55.495776 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.495991 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20240502 14:10:55.499073 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.497743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44562)
0502 14:10:55.498363 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.498550 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20240502 14:10:55.499513 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
W20240502 14:10:55.544044 13562 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:10:55.544880 13562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:55.545219 13562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:55.545780 13562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:55.576131 13562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:55.576434 13562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:55.576740 13562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:55.576996 13562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:55.577260 13562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:55.614719 13562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:42921,127.27.174.125:40715,127.27.174.124:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:55.616209 13562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:55.617341 13562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:55.631763 13570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:55.642890 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 4
I20240502 14:10:55.645184 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.644036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44564)
0502 14:10:55.644610 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.644765 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20240502 14:10:55.648207 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.646475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44580)
0502 14:10:55.647353 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.647590 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20240502 14:10:55.648725 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.874256 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } attempt: 7
May 02 14:10:55 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:10:55.904009 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.876181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.27.174.125:40715)
0502 14:10:55.880077 (+ 3896us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:55.880091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:55.880222 (+ 131us) server_negotiation.cc:407] Connection header received
0502 14:10:55.880752 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:55.880767 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:55.880919 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:55.881054 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:55.883862 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.884651 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.886913 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.887246 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.893726 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:55.893806 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:55.893860 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:55.893987 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:55.897455 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.899255 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.899277 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.899307 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.899444 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:55.900249 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:55.900268 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:55.900280 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:55.900645 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:55.900905 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:55.902427 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0502 14:10:55.902721 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3483,"thread_start_us":3235,"threads_started":1}
I20240502 14:10:55.905987 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.875864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40842)
0502 14:10:55.879970 (+ 4106us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.880048 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:55.880483 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:55.881226 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:55.881267 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:55.882559 (+ 1292us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:55.883451 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.883549 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.884918 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:55.884968 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:55.886572 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:55.886652 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:55.887128 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:55.888127 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:55.888247 (+ 120us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:55.893382 (+ 5135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:55.897721 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.897769 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.897808 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.899039 (+ 1231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.899667 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:55.899709 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:55.899744 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:55.900031 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:55.900894 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:55.900940 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:55.901359 (+ 419us) client_negotiation.cc:769] Sending connection context
0502 14:10:55.901792 (+ 433us) client_negotiation.cc:240] Negotiation successful
0502 14:10:55.902558 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3128,"thread_start_us":2645,"threads_started":1}
I20240502 14:10:55.907866 13477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:10:55.911542 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.909462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44596)
0502 14:10:55.910453 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.910751 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20240502 14:10:55.916388 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.914319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44598)
0502 14:10:55.915315 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.915618 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20240502 14:10:55.916955 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.948158 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 5
I20240502 14:10:55.950937 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.949719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44610)
0502 14:10:55.950339 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.950521 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20240502 14:10:55.953300 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.952256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44618)
0502 14:10:55.952785 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.952921 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20240502 14:10:55.953691 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:55.975338 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 1
I20240502 14:10:55.977916 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.976786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44620)
0502 14:10:55.977358 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.977514 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20240502 14:10:55.980513 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:55.979517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44624)
0502 14:10:55.980007 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:55.980145 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20240502 14:10:55.980901 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:56.062264 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 2
I20240502 14:10:56.064903 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.063757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44636)
0502 14:10:56.064339 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.064498 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20240502 14:10:56.067541 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.066472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44638)
0502 14:10:56.066970 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.067100 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20240502 14:10:56.067930 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:56.153545 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 3
I20240502 14:10:56.155721 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.154603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44642)
0502 14:10:56.155183 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.155335 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20240502 14:10:56.158382 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.157334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44646)
0502 14:10:56.157824 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.157965 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20240502 14:10:56.158807 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:56.299443 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 4
I20240502 14:10:56.302250 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.301040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44662)
0502 14:10:56.301620 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.301771 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20240502 14:10:56.304771 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.303715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44670)
0502 14:10:56.304243 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.304386 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20240502 14:10:56.305207 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:56.501343 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 6
I20240502 14:10:56.505424 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.502578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44676)
0502 14:10:56.504695 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.504874 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1648,"thread_start_us":1396,"threads_started":1}
I20240502 14:10:56.509130 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.508078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44684)
0502 14:10:56.508599 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.508751 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:10:56.509616 13555 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
I20240502 14:10:56.588543 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 5
I20240502 14:10:56.590845 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.589689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44690)
0502 14:10:56.590257 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.590417 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20240502 14:10:56.593024 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.592016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44706)
0502 14:10:56.592488 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.592633 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20240502 14:10:56.593436 13477 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:34141: Network error: Client connection negotiation failed: client connection to 127.27.174.124:34141: connect: Connection refused (error 111)
W20240502 14:10:55.632601 13569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:55.633872 13562 server_base.cc:1034] running on GCE node
W20240502 14:10:55.633826 13572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:56.714284 13562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:10:56.716782 13562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:10:56.718116 13562 hybrid_clock.cc:648] HybridClock initialized: now 1714659056718092 us; error 40 us; skew 500 ppm
I20240502 14:10:56.718890 13562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659056, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:10:56.724625 13562 init.cc:377] Logged in from keytab as kudu/127.27.174.124@KRBTEST.COM (short username kudu)
I20240502 14:10:56.729851 13562 webserver.cc:460] Webserver started at http://127.27.174.124:44089/ using document root <none> and password file <none>
I20240502 14:10:56.730800 13562 fs_manager.cc:359] Metadata directory not provided
I20240502 14:10:56.730991 13562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:10:56.731417 13562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:10:56.736374 13562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data/instance:
uuid: "655bc93fbe9547a285b39b795217f1b1"
format_stamp: "Formatted at 2024-05-02 14:10:56 on dist-test-slave-dlb8"
server_key: "7ce6cd770b4555de08944738f0c0eea7"
server_key_iv: "f8a4fdf8317191a9ca35454237223c21"
server_key_version: "encryptionkey@0"
I20240502 14:10:56.737727 13562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal/instance:
uuid: "655bc93fbe9547a285b39b795217f1b1"
format_stamp: "Formatted at 2024-05-02 14:10:56 on dist-test-slave-dlb8"
server_key: "7ce6cd770b4555de08944738f0c0eea7"
server_key_iv: "f8a4fdf8317191a9ca35454237223c21"
server_key_version: "encryptionkey@0"
I20240502 14:10:56.745668 13562 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20240502 14:10:56.751066 13585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:10:56.752027 13562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:10:56.752344 13562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal
uuid: "655bc93fbe9547a285b39b795217f1b1"
format_stamp: "Formatted at 2024-05-02 14:10:56 on dist-test-slave-dlb8"
server_key: "7ce6cd770b4555de08944738f0c0eea7"
server_key_iv: "f8a4fdf8317191a9ca35454237223c21"
server_key_version: "encryptionkey@0"
I20240502 14:10:56.752662 13562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:10:56.800956 13562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:10:56.806550 13562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:10:56.806978 13562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:10:56.874081 13562 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:34141
I20240502 14:10:56.874169 13636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:34141 every 8 connection(s)
I20240502 14:10:56.876621 13562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
I20240502 14:10:56.881289 13637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:10:56.883562 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13562
I20240502 14:10:56.884006 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714658657003751-28345-0/minicluster-data/master-2/wal/instance
I20240502 14:10:56.885146 28345 external_mini_cluster.cc:1428] Setting key 56cce75d216f7ff422be6d12daeac48d
I20240502 14:10:56.893833 13637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659056, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:56.921204 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.887479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.27.174.124:34141)
0502 14:10:56.890327 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.890353 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.890403 (+ 50us) server_negotiation.cc:407] Connection header received
0502 14:10:56.894741 (+ 4338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.894833 (+ 92us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.895186 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.895493 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.899044 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.901054 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.903071 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.903565 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.909670 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.909842 (+ 172us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.909865 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.909962 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.914819 (+ 4857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.915831 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.915857 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.915881 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.916108 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.916636 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.916664 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.916686 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.917467 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.917923 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.918426 (+ 503us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.918972 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2251,"thread_start_us":1784,"threads_started":1}
I20240502 14:10:56.925354 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.899320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.27.174.126:42921)
0502 14:10:56.901584 (+ 2264us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.901613 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.903238 (+ 1625us) server_negotiation.cc:407] Connection header received
0502 14:10:56.903611 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.903637 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.903858 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.904031 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.907004 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.908346 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.910085 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.910602 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.914797 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.914930 (+ 133us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.914967 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.915111 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.919685 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.920638 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.920667 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.920685 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.920859 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.921374 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.921401 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.921426 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.921929 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.922197 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.922992 (+ 795us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.923421 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":1141,"threads_started":1}
I20240502 14:10:56.925365 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.899108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:42921 (local address 127.0.0.1:34290)
0502 14:10:56.903002 (+ 3894us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.903096 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:56.903376 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:56.904191 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:56.904219 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:56.905626 (+ 1407us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:56.906703 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.906775 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.908565 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.908594 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:56.909843 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.909890 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.910138 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.910840 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:56.910920 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:56.914539 (+ 3619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:56.919872 (+ 5333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.919895 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.919909 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.920445 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.921015 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.921031 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.921044 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.921185 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.922166 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:56.922197 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:56.922475 (+ 278us) client_negotiation.cc:769] Sending connection context
0502 14:10:56.922733 (+ 258us) client_negotiation.cc:240] Negotiation successful
0502 14:10:56.923073 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3241,"thread_start_us":2370,"threads_started":1}
I20240502 14:10:56.927716 13637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659056, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:10:56.948772 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.924936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.27.174.125:40715)
0502 14:10:56.927587 (+ 2651us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.927606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.927649 (+ 43us) server_negotiation.cc:407] Connection header received
0502 14:10:56.927785 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.927810 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.928011 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.928204 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.930489 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.931735 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.933164 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.933667 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.938489 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.938587 (+ 98us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.938607 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.938725 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.942946 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.944228 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.944259 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.944283 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.944465 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.944990 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.945021 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.945041 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.945500 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.945904 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.946301 (+ 397us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.946611 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2217,"thread_start_us":1971,"threads_started":1}
I20240502 14:10:56.949115 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.929218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.27.174.125:40715)
0502 14:10:56.931022 (+ 1804us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.931047 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.931089 (+ 42us) server_negotiation.cc:407] Connection header received
0502 14:10:56.931210 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.931233 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.931423 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.931584 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.932920 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.934186 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.935875 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.936215 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.939450 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.939518 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.939542 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.939645 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.943861 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.945033 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.945054 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.945069 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.945307 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.945933 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.945960 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.945981 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.946440 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.946660 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.947378 (+ 718us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.947664 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"spinlock_wait_cycles":18816,"thread_start_us":990,"threads_started":1}
I20240502 14:10:56.949615 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.928976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:40715 (local address 127.0.0.1:40856)
0502 14:10:56.929461 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.929488 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:56.929703 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:56.931642 (+ 1939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:56.931660 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:56.932248 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:56.932670 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.932717 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.934401 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.934434 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:56.935505 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.935566 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.935874 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.936662 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:56.936764 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:56.939196 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:56.944099 (+ 4903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.944130 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.944156 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.944851 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.945503 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.945528 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.945548 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.945747 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.946655 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:56.946681 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:56.946953 (+ 272us) client_negotiation.cc:769] Sending connection context
0502 14:10:56.947230 (+ 277us) client_negotiation.cc:240] Negotiation successful
0502 14:10:56.947640 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20240502 14:10:56.951368 13637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659056, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
May 02 14:10:56 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:10:56.971570 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.952648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44718)
0502 14:10:56.953184 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:56.953221 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:56.953434 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:56.954398 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:56.954429 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:56.955205 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:56.955671 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.955719 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.956923 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.956945 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:56.957747 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:56.957798 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.958091 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.958884 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:56.958979 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:56.961857 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:56.966207 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.966241 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.966274 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.967041 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.967514 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:56.967541 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:56.967567 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:56.967763 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:56.968616 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:56.968636 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:56.968815 (+ 179us) client_negotiation.cc:769] Sending connection context
0502 14:10:56.969086 (+ 271us) client_negotiation.cc:240] Negotiation successful
0502 14:10:56.969466 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"spinlock_wait_cycles":27392}
I20240502 14:10:56.971648 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.952884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.27.174.124:34141)
0502 14:10:56.953642 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.953665 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.953708 (+ 43us) server_negotiation.cc:407] Connection header received
0502 14:10:56.953834 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.953865 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.954119 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.954279 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.955890 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.956739 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.958013 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.958562 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.962133 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.962232 (+ 99us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.962259 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.962386 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.965984 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.967204 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.967223 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.967236 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.967359 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.967925 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.967947 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.967966 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.968438 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.968686 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.969235 (+ 549us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.969766 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":63}
I20240502 14:10:56.973285 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:56.951777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.27.174.126:42921)
0502 14:10:56.952357 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:56.952390 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:56.952599 (+ 209us) server_negotiation.cc:407] Connection header received
0502 14:10:56.953498 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:56.953526 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:56.953777 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:56.953968 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:56.956353 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.957246 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:56.958760 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:56.959589 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:56.963652 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:56.963726 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:56.963744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:56.963837 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:56.967730 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.968934 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.968963 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.968983 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.969152 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:56.969733 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:56.969761 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:56.969783 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:56.970264 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:56.970657 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:56.971177 (+ 520us) server_negotiation.cc:299] Negotiation successful
0502 14:10:56.971601 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20240502 14:10:56.979449 13637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: Bootstrap starting.
I20240502 14:10:56.980249 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:10:56.984531 13637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:56.986212 13637 log.cc:824] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:56.991323 13637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: No bootstrap required, opened a new log
I20240502 14:10:56.988701 13452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cff9edb101b04591bd0797953b7e672b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34300
I20240502 14:10:57.019626 13637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.020448 13637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655bc93fbe9547a285b39b795217f1b1, State: Initialized, Role: FOLLOWER
I20240502 14:10:57.021124 13637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.023844 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.024502 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.025826 13637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: configured and running, proceeding with master startup.
W20240502 14:10:57.050487 13657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:10:57.050849 13657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:10:57.061736 13591 tablet.cc:2369] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:10:57.047436 13562 master_runner.cc:186] Error getting master registration for 127.27.174.126:42921: OK, instance_id { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" instance_seqno: 1714659053492614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 14:10:57.068620 13562 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 14:10:57.145879 13477 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 6
May 02 14:10:57 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20240502 14:10:57.169734 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.147330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.27.174.124:34141)
0502 14:10:57.148006 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.148028 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.149261 (+ 1233us) server_negotiation.cc:407] Connection header received
0502 14:10:57.149575 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.149617 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.149836 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.150029 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.151853 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.152990 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.154550 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.155099 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.159460 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:57.159568 (+ 108us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:57.159595 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:57.159733 (+ 138us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:57.163556 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.164837 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.164860 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.164874 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.165004 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.165658 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.165698 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.165725 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.166320 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:57.166557 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:57.167400 (+ 843us) server_negotiation.cc:299] Negotiation successful
0502 14:10:57.167820 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20240502 14:10:57.169943 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.147163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44726)
0502 14:10:57.149118 (+ 1955us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:57.149152 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:57.149369 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:57.150122 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:57.150149 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:57.150971 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:57.151551 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:57.151597 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.153181 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.153208 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:57.154288 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:57.154339 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.154650 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.155477 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:57.155569 (+ 92us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:57.159205 (+ 3636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:57.163748 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:57.163794 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:57.163818 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:57.164658 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:57.165197 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:57.165224 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:57.165246 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:57.165457 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:57.166532 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:57.166567 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:57.166851 (+ 284us) client_negotiation.cc:769] Sending connection context
0502 14:10:57.167163 (+ 312us) client_negotiation.cc:240] Negotiation successful
0502 14:10:57.167665 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"thread_start_us":1292,"threads_started":1}
I20240502 14:10:57.179942 13477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: Bootstrap starting.
I20240502 14:10:57.187173 13477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: Neither blocks nor log segments found. Creating new log.
I20240502 14:10:57.189535 13477 log.cc:824] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:10:57.196097 13477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: No bootstrap required, opened a new log
I20240502 14:10:57.215217 13477 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.216066 13477 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff9edb101b04591bd0797953b7e672b, State: Initialized, Role: FOLLOWER
I20240502 14:10:57.216840 13477 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.219452 13661 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.220260 13661 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.221351 13477 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:57.221446 13452 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:10:57.221796 13452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:57.228085 13452 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.231859 13612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cff9edb101b04591bd0797953b7e672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "655bc93fbe9547a285b39b795217f1b1"
I20240502 14:10:57.232461 13612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:57.237157 13612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff9edb101b04591bd0797953b7e672b in term 1.
I20240502 14:10:57.238170 13452 leader_election.cc:288] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [CANDIDATE]: Term 1 election: Requested vote from peers d5dee1ab184d44c99f4468b974a6def5 (127.27.174.125:40715), 655bc93fbe9547a285b39b795217f1b1 (127.27.174.124:34141)
I20240502 14:10:57.238447 13429 leader_election.cc:302] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655bc93fbe9547a285b39b795217f1b1, cff9edb101b04591bd0797953b7e672b; no voters:
I20240502 14:10:57.239365 13661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:10:57.254520 13661 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [term 1 LEADER]: Becoming Leader. State: Replica: cff9edb101b04591bd0797953b7e672b, State: Running, Role: LEADER
I20240502 14:10:57.256875 13661 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.285530 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff9edb101b04591bd0797953b7e672b. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.286365 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: This master's current role is: LEADER
W20240502 14:10:57.311097 13680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:10:57.311437 13680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:10:57.311954 13681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:10:57.320138 13681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:10:57.346578 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.271272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.27.174.125:40715)
0502 14:10:57.272319 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.272348 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.276605 (+ 4257us) server_negotiation.cc:407] Connection header received
0502 14:10:57.282188 (+ 5583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.282226 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.282515 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.282687 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.304192 (+ 21505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.305426 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.310852 (+ 5426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.311392 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.328071 (+ 16679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:57.328177 (+ 106us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:57.328220 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:57.328348 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:57.335589 (+ 7241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.339186 (+ 3597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.339227 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.339249 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.339428 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.340463 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.340494 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.340515 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.341494 (+ 979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:57.341875 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:57.344073 (+ 2198us) server_negotiation.cc:299] Negotiation successful
0502 14:10:57.344545 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20240502 14:10:57.290136 13408 master_runner.cc:186] Error getting master registration for 127.27.174.125:40715: OK, instance_id { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" instance_seqno: 1714659055165593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 14:10:57.356177 13408 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 14:10:57.356252 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.272958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.27.174.126:42921)
0502 14:10:57.273781 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.273810 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.291841 (+ 18031us) server_negotiation.cc:407] Connection header received
0502 14:10:57.300700 (+ 8859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.300738 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.300998 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.301163 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.312333 (+ 11170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.313444 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.319366 (+ 5922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.319866 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.326601 (+ 6735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:57.326691 (+ 90us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:57.326716 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:57.326846 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:57.337779 (+ 10933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.350764 (+ 12985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.350792 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.350820 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.351042 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.351794 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.351822 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.351847 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.352421 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:57.352760 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:57.353731 (+ 971us) server_negotiation.cc:299] Negotiation successful
0502 14:10:57.354251 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:10:57.357069 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.296208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.27.174.124:34141)
0502 14:10:57.296996 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.297021 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.309616 (+ 12595us) server_negotiation.cc:407] Connection header received
0502 14:10:57.309939 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.309971 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.310219 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.310385 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.326306 (+ 15921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.327475 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.330832 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.331347 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.338628 (+ 7281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:57.338729 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:57.338758 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:57.338895 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:57.345259 (+ 6364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.347745 (+ 2486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.347810 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.347859 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.348158 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.349431 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.349458 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.349520 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.350426 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:57.350997 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:57.352857 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0502 14:10:57.353623 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:10:57.372565 13612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [term 1 FOLLOWER]: Refusing update from remote peer cff9edb101b04591bd0797953b7e672b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:10:57.373797 13669 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:10:57.412119 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff9edb101b04591bd0797953b7e672b. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.412945 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.418406 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.422427 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.438235 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.439924 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [sys.catalog]: This master's current role is: LEADER
I20240502 14:10:57.440712 13681 catalog_manager.cc:1347] Generated new cluster ID: 5ad2f00b246e465b8753a5c8d42a1e3b
I20240502 14:10:57.440974 13681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:10:57.485446 13681 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:10:57.487699 13681 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:10:57.523254 13681 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b: Generated new TSK 0
I20240502 14:10:57.524803 13681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:10:57.570117 13555 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } attempt: 7
May 02 14:10:57 dist-test-slave-dlb8 krb5kdc[13395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20240502 14:10:57.671007 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.572720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.27.174.124:34141)
0502 14:10:57.573556 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.573589 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.593579 (+ 19990us) server_negotiation.cc:407] Connection header received
0502 14:10:57.599207 (+ 5628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.599240 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.599458 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.599598 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.601854 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.603062 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.605635 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.606211 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.620730 (+ 14519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:10:57.620842 (+ 112us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:10:57.620869 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:10:57.621006 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:10:57.627990 (+ 6984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.631976 (+ 3986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.632108 (+ 132us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.632185 (+ 77us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.632879 (+ 694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:10:57.635667 (+ 2788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:10:57.635746 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:10:57.635806 (+ 60us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:10:57.638023 (+ 2217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:10:57.640216 (+ 2193us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:57.643007 (+ 2791us) server_negotiation.cc:299] Negotiation successful
0502 14:10:57.648526 (+ 5519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20240502 14:10:57.671589 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:57.571411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:34141 (local address 127.0.0.1:44752)
0502 14:10:57.593363 (+ 21952us) negotiation.cc:107] Waiting for socket to connect
0502 14:10:57.593417 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0502 14:10:57.598914 (+ 5497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:10:57.600094 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:10:57.600121 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:10:57.600968 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:10:57.601520 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:57.601570 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.603418 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:10:57.603507 (+ 89us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:10:57.604636 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:10:57.604692 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:10:57.612369 (+ 7677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:57.613366 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:10:57.613461 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:10:57.620143 (+ 6682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:10:57.628409 (+ 8266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:57.628531 (+ 122us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:57.628604 (+ 73us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:57.631242 (+ 2638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:57.633929 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:10:57.634077 (+ 148us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:10:57.634100 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:10:57.635030 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:10:57.639020 (+ 3990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:10:57.639103 (+ 83us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:10:57.640795 (+ 1692us) client_negotiation.cc:769] Sending connection context
0502 14:10:57.643402 (+ 2607us) client_negotiation.cc:240] Negotiation successful
0502 14:10:57.648669 (+ 5267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21320,"thread_start_us":26908,"threads_started":1}
I20240502 14:10:57.685528 13555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: Bootstrap starting.
W20240502 14:10:57.708969 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.665847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.27.174.125:40715)
0502 14:10:57.666583 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.666610 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.697891 (+ 31281us) server_negotiation.cc:407] Connection header received
0502 14:10:57.705405 (+ 7514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.705434 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.705659 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.705822 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.708020 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53462: BlockingRecv error: recv got EOF from 127.0.0.1:53462 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20240502 14:10:57.711639 13555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: Neither blocks nor log segments found. Creating new log.
W20240502 14:10:57.713831 13641 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.673953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.27.174.126:42921)
0502 14:10:57.674928 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.674959 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.703221 (+ 28262us) server_negotiation.cc:407] Connection header received
0502 14:10:57.704074 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.704107 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.704347 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.704512 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.712915 (+ 8403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41472: BlockingRecv error: recv got EOF from 127.0.0.1:41472 (error 108)
Metrics: {"server-negotiator.queue_time_us":375}
I20240502 14:10:57.716836 13555 log.cc:824] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: Log is configured to *not* fsync() on all Append() calls
W20240502 14:10:57.723875 13640 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.699805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.27.174.124:34141)
0502 14:10:57.700417 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.700442 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.719132 (+ 18690us) server_negotiation.cc:407] Connection header received
0502 14:10:57.719535 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.719574 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.719846 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.720030 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.722894 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51288: BlockingRecv error: recv got EOF from 127.0.0.1:51288 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20240502 14:10:57.749862 13555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: No bootstrap required, opened a new log
W20240502 14:10:57.776644 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.769629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.27.174.125:40715)
0502 14:10:57.770332 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.770358 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.771613 (+ 1255us) server_negotiation.cc:407] Connection header received
0502 14:10:57.772469 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.772498 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.772779 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.772947 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.775725 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53468: BlockingRecv error: recv got EOF from 127.0.0.1:53468 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20240502 14:10:57.789852 13555 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.791648 13555 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dee1ab184d44c99f4468b974a6def5, State: Initialized, Role: FOLLOWER
I20240502 14:10:57.792593 13555 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } }
I20240502 14:10:57.801169 13707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.804908 13555 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:10:57.805446 13529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:10:57.808691 13707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.810317 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cff9edb101b04591bd0797953b7e672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5dee1ab184d44c99f4468b974a6def5"
I20240502 14:10:57.816365 13529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [term 1 FOLLOWER]: Refusing update from remote peer cff9edb101b04591bd0797953b7e672b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:10:57.825793 13530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff9edb101b04591bd0797953b7e672b in term 1.
I20240502 14:10:57.831312 13661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cff9edb101b04591bd0797953b7e672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:10:57.874234 13509 tablet.cc:2369] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:10:57.880128 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.846634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.27.174.125:40715)
0502 14:10:57.847530 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.847562 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.871124 (+ 23562us) server_negotiation.cc:407] Connection header received
0502 14:10:57.871643 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.871674 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.871963 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.872173 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.879192 (+ 7019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53474: BlockingRecv error: recv got EOF from 127.0.0.1:53474 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20240502 14:10:57.925464 13707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff9edb101b04591bd0797953b7e672b. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.926429 13707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:10:57.928056 13723 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cff9edb101b04591bd0797953b7e672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff9edb101b04591bd0797953b7e672b" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 42921 } } peers { permanent_uuid: "d5dee1ab184d44c99f4468b974a6def5" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 40715 } } peers { permanent_uuid: "655bc93fbe9547a285b39b795217f1b1" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 34141 } } }
I20240502 14:10:57.929217 13723 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 14:10:57.938227 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:10:57.921088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.27.174.125:40715)
0502 14:10:57.921996 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:57.922047 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:57.933404 (+ 11357us) server_negotiation.cc:407] Connection header received
0502 14:10:57.935136 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:57.935156 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:57.935382 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:57.935518 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:10:57.937347 (+ 1829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53486: BlockingRecv error: recv got EOF from 127.0.0.1:53486 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:10:57.961625 13732 catalog_manager.cc:1259] Loaded cluster ID: 5ad2f00b246e465b8753a5c8d42a1e3b
I20240502 14:10:57.963497 13732 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: loading cluster ID for follower catalog manager: success
I20240502 14:10:57.972646 13732 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: acquiring CA information for follower catalog manager: success
I20240502 14:10:57.991189 13732 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5dee1ab184d44c99f4468b974a6def5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:10:58.035943 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:58.008144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.27.174.125:40715)
0502 14:10:58.008862 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:58.008885 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:58.023616 (+ 14731us) server_negotiation.cc:407] Connection header received
0502 14:10:58.024226 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:58.024256 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:58.024599 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:58.024789 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:10:58.027648 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.028967 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:58.031048 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.031584 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:58.031813 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:10:58.032504 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:10:58.032687 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:58.034157 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0502 14:10:58.034349 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20240502 14:10:58.056123 13657 catalog_manager.cc:1259] Loaded cluster ID: 5ad2f00b246e465b8753a5c8d42a1e3b
I20240502 14:10:58.056385 13657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: loading cluster ID for follower catalog manager: success
I20240502 14:10:58.066421 13657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: acquiring CA information for follower catalog manager: success
I20240502 14:10:58.071249 13657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 655bc93fbe9547a285b39b795217f1b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:10:58.093912 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:58.076007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.27.174.124:34141)
0502 14:10:58.076650 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:58.076681 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:58.083603 (+ 6922us) server_negotiation.cc:407] Connection header received
0502 14:10:58.084641 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:58.084672 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:58.084988 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:58.085146 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:10:58.086624 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.087895 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:58.089902 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.090428 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:58.091125 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:10:58.091807 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:10:58.091960 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:58.092409 (+ 449us) server_negotiation.cc:299] Negotiation successful
0502 14:10:58.092567 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20240502 14:10:58.173772 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:58.144136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.27.174.126:42921)
0502 14:10:58.144858 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:58.144889 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:58.155388 (+ 10499us) server_negotiation.cc:407] Connection header received
0502 14:10:58.155826 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:58.155854 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:58.156171 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:58.156357 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:10:58.160329 (+ 3972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.161622 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:58.165947 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.166481 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:58.166710 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:10:58.167252 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:10:58.167416 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:58.172170 (+ 4754us) server_negotiation.cc:299] Negotiation successful
0502 14:10:58.172358 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20240502 14:10:58.175992 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:58.137719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.27.174.125:40715)
0502 14:10:58.138428 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:58.138453 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:58.153653 (+ 15200us) server_negotiation.cc:407] Connection header received
0502 14:10:58.154135 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:58.154166 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:58.154505 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:58.154665 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:10:58.161991 (+ 7326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.163357 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:58.167869 (+ 4512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.168432 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:58.168608 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:10:58.168879 (+ 271us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:10:58.169030 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:58.174382 (+ 5352us) server_negotiation.cc:299] Negotiation successful
0502 14:10:58.174565 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20240502 14:10:58.177222 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:10:58.157162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.27.174.124:34141)
0502 14:10:58.157803 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0502 14:10:58.157828 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:10:58.159151 (+ 1323us) server_negotiation.cc:407] Connection header received
0502 14:10:58.162071 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:10:58.162100 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:10:58.162401 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:10:58.162545 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:10:58.168329 (+ 5784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.169602 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:10:58.173730 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:10:58.174307 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:10:58.174975 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:10:58.175284 (+ 309us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:10:58.175393 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0502 14:10:58.175775 (+ 382us) server_negotiation.cc:299] Negotiation successful
0502 14:10:58.175923 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20240502 14:10:58.184896 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13408
I20240502 14:10:58.230551 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13484
I20240502 14:10:58.265334 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13562
2024-05-02T14:10:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (7531 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 02 14:10:58 dist-test-slave-dlb8 krb5kdc[13753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 02 14:10:58 dist-test-slave-dlb8 krb5kdc[13753](info): set up 2 sockets
May 02 14:10:58 dist-test-slave-dlb8 krb5kdc[13753](info): commencing operation
krb5kdc: starting...
W20240502 14:10:59.265303 28345 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/test-user.keytab.
May 02 14:10:59 dist-test-slave-dlb8 krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-02T14:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
Entry for principal kudu/127.27.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab.
I20240502 14:10:59.476703 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41141
--webserver_interface=127.27.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240502 14:10:59.765002 13766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:10:59.765828 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:10:59.766191 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:10:59.766803 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:10:59.795250 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:10:59.795569 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:10:59.795852 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:10:59.796077 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:10:59.796293 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:10:59.829515 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.126.keytab
--principal=kudu/127.27.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.126:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:10:59.830670 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:10:59.831681 13766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:10:59.847208 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:10:59.848773 13766 server_base.cc:1034] running on GCE node
W20240502 14:10:59.847285 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:10:59.847419 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:00.893606 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:11:00.896178 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:11:00.897681 13766 hybrid_clock.cc:648] HybridClock initialized: now 1714659060897578 us; error 74 us; skew 500 ppm
I20240502 14:11:00.899330 13766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:11:00 dist-test-slave-dlb8 krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:11:00.906396 13766 init.cc:377] Logged in from keytab as kudu/127.27.174.126@KRBTEST.COM (short username kudu)
I20240502 14:11:00.911134 13766 webserver.cc:460] Webserver started at http://127.27.174.126:32997/ using document root <none> and password file <none>
I20240502 14:11:00.911926 13766 fs_manager.cc:359] Metadata directory not provided
I20240502 14:11:00.912091 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:11:00.912462 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:11:00.917464 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data/instance:
uuid: "6148c0bc14fd40f6a81369ca9948cff7"
format_stamp: "Formatted at 2024-05-02 14:11:00 on dist-test-slave-dlb8"
server_key: "da275707ef9d48426828e4b17e85f2d8"
server_key_iv: "5fe5141491bd5837f0514631afc70d09"
server_key_version: "encryptionkey@0"
I20240502 14:11:00.918612 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance:
uuid: "6148c0bc14fd40f6a81369ca9948cff7"
format_stamp: "Formatted at 2024-05-02 14:11:00 on dist-test-slave-dlb8"
server_key: "da275707ef9d48426828e4b17e85f2d8"
server_key_iv: "5fe5141491bd5837f0514631afc70d09"
server_key_version: "encryptionkey@0"
I20240502 14:11:00.925494 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:11:00.930764 13783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:11:00.931685 13766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 14:11:00.931988 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal
uuid: "6148c0bc14fd40f6a81369ca9948cff7"
format_stamp: "Formatted at 2024-05-02 14:11:00 on dist-test-slave-dlb8"
server_key: "da275707ef9d48426828e4b17e85f2d8"
server_key_iv: "5fe5141491bd5837f0514631afc70d09"
server_key_version: "encryptionkey@0"
I20240502 14:11:00.932296 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:11:00.982923 13766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:11:00.988304 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:11:00.988723 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:11:01.053972 13766 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.126:41141
I20240502 14:11:01.054098 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.126:41141 every 8 connection(s)
I20240502 14:11:01.056608 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/data/info.pb
I20240502 14:11:01.061609 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:11:01.062351 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13766
I20240502 14:11:01.062770 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-0/wal/instance
I20240502 14:11:01.063724 28345 external_mini_cluster.cc:1428] Setting key f00d7d2dc5b762684202ce9b54afd8f2
I20240502 14:11:01.073642 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.125@KRBTEST.COM" created.
May 02 14:11:01 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:11:01.112066 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.081873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.27.174.126:41141)
0502 14:11:01.085878 (+ 4005us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:01.085906 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:01.085970 (+ 64us) server_negotiation.cc:407] Connection header received
0502 14:11:01.088627 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:01.088651 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:01.088906 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:01.089088 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:01.091991 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:01.093481 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:01.094775 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:01.095448 (+ 673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:01.098843 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:01.098991 (+ 148us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:01.099034 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:01.099178 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:01.105144 (+ 5966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:01.106268 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:01.106305 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:01.106345 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:01.106585 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:01.107092 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:01.107109 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:01.107121 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:01.107687 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:01.107951 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:01.108698 (+ 747us) server_negotiation.cc:299] Negotiation successful
0502 14:11:01.109324 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":3468,"thread_start_us":2699,"threads_started":1}
I20240502 14:11:01.112181 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.079003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41141 (local address 127.0.0.1:44554)
0502 14:11:01.081820 (+ 2817us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.081931 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:01.088021 (+ 6090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:01.089257 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:01.089307 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:01.090503 (+ 1196us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:01.091578 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:01.091647 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:01.093724 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:01.093756 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:01.094539 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:01.094573 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:01.094856 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:01.095712 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:01.095776 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:01.098589 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:01.105374 (+ 6785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:01.105402 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:01.105429 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:01.106016 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:01.106733 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:01.106753 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:01.106769 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:01.106946 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:01.107851 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:01.107889 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:01.108145 (+ 256us) client_negotiation.cc:769] Sending connection context
0502 14:11:01.108439 (+ 294us) client_negotiation.cc:240] Negotiation successful
0502 14:11:01.109018 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"thread_start_us":1774,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
Entry for principal kudu/127.27.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab.
I20240502 14:11:01.115135 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:11:01.115651 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:34987
--webserver_interface=127.27.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240502 14:11:01.117323 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.116293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59126)
0502 14:11:01.116815 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.116971 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20240502 14:11:01.117822 13787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:01.121958 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.120947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59136)
0502 14:11:01.121441 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.121580 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:11:01.122367 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:01.171907 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 1
I20240502 14:11:01.174155 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.173033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59144)
0502 14:11:01.173555 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.173723 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20240502 14:11:01.176334 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.175360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59150)
0502 14:11:01.175818 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.175968 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20240502 14:11:01.176707 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:01.245133 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 2
I20240502 14:11:01.247344 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.246291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59162)
0502 14:11:01.246808 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.246948 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20240502 14:11:01.249513 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.248543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59168)
0502 14:11:01.249033 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.249162 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20240502 14:11:01.249838 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:01.341326 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 3
I20240502 14:11:01.343711 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.342527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59178)
0502 14:11:01.343094 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.343256 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20240502 14:11:01.345816 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.344805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59184)
0502 14:11:01.345309 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.345443 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20240502 14:11:01.346132 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
W20240502 14:11:01.405104 13842 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:11:01.406066 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:11:01.406476 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:11:01.407088 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:11:01.436151 13842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:11:01.436517 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:11:01.436796 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:11:01.437024 13842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:11:01.437252 13842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:11:01.470552 13842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.125.keytab
--principal=kudu/127.27.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.125:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:11:01.471699 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:11:01.472681 13842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:11:01.487354 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:01.489524 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 4
I20240502 14:11:01.492338 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.490483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59192)
0502 14:11:01.491414 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.491654 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20240502 14:11:01.495127 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.493494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59206)
0502 14:11:01.494263 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.494498 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20240502 14:11:01.495577 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:01.795012 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 5
I20240502 14:11:01.797719 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.796640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59214)
0502 14:11:01.797177 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.797324 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20240502 14:11:01.799710 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:01.798805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59226)
0502 14:11:01.799255 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:01.799373 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:11:01.800071 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
I20240502 14:11:02.347667 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 6
I20240502 14:11:02.351555 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.348840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59228)
0502 14:11:02.350870 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.351025 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1558,"thread_start_us":1316,"threads_started":1}
I20240502 14:11:02.362293 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.361220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59238)
0502 14:11:02.361714 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.361873 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20240502 14:11:02.362776 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.125:34987: Network error: Client connection negotiation failed: client connection to 127.27.174.125:34987: connect: Connection refused (error 111)
W20240502 14:11:01.487345 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:11:01.488332 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:01.488880 13842 server_base.cc:1034] running on GCE node
I20240502 14:11:02.532248 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:11:02.534940 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:11:02.536280 13842 hybrid_clock.cc:648] HybridClock initialized: now 1714659062536256 us; error 46 us; skew 500 ppm
I20240502 14:11:02.537046 13842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:11:02 dist-test-slave-dlb8 krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:11:02.542838 13842 init.cc:377] Logged in from keytab as kudu/127.27.174.125@KRBTEST.COM (short username kudu)
I20240502 14:11:02.547940 13842 webserver.cc:460] Webserver started at http://127.27.174.125:40417/ using document root <none> and password file <none>
I20240502 14:11:02.548796 13842 fs_manager.cc:359] Metadata directory not provided
I20240502 14:11:02.548992 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:11:02.549429 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:11:02.554539 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data/instance:
uuid: "245151308cf44e509d776584225ff968"
format_stamp: "Formatted at 2024-05-02 14:11:02 on dist-test-slave-dlb8"
server_key: "1bb490a88b643e419ac8cccc7c9e7492"
server_key_iv: "4b5ee28a6df1d06130816e1a1837ca35"
server_key_version: "encryptionkey@0"
I20240502 14:11:02.555651 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal/instance:
uuid: "245151308cf44e509d776584225ff968"
format_stamp: "Formatted at 2024-05-02 14:11:02 on dist-test-slave-dlb8"
server_key: "1bb490a88b643e419ac8cccc7c9e7492"
server_key_iv: "4b5ee28a6df1d06130816e1a1837ca35"
server_key_version: "encryptionkey@0"
I20240502 14:11:02.562685 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240502 14:11:02.568166 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:11:02.569166 13842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:11:02.569489 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal
uuid: "245151308cf44e509d776584225ff968"
format_stamp: "Formatted at 2024-05-02 14:11:02 on dist-test-slave-dlb8"
server_key: "1bb490a88b643e419ac8cccc7c9e7492"
server_key_iv: "4b5ee28a6df1d06130816e1a1837ca35"
server_key_version: "encryptionkey@0"
I20240502 14:11:02.569835 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:11:02.635130 13842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:11:02.640411 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:11:02.640856 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:11:02.711607 13842 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.125:34987
I20240502 14:11:02.711697 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.125:34987 every 8 connection(s)
I20240502 14:11:02.714170 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/data/info.pb
I20240502 14:11:02.719067 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13842
I20240502 14:11:02.719139 13913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:11:02.719522 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-1/wal/instance
I20240502 14:11:02.720476 28345 external_mini_cluster.cc:1428] Setting key 319eba82a14e146bb0e2e6e656b45eb8
I20240502 14:11:02.732334 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.174.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.174.124@KRBTEST.COM" created.
May 02 14:11:02 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:11:02.768656 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.738027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.27.174.126:41141)
0502 14:11:02.740245 (+ 2218us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:02.740284 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:02.741258 (+ 974us) server_negotiation.cc:407] Connection header received
0502 14:11:02.746758 (+ 5500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:02.746782 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:02.746956 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:02.747102 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:02.750511 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.751391 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:02.752827 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.753424 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:02.757164 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:02.757303 (+ 139us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:02.757343 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:02.757498 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:02.763200 (+ 5702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:02.764295 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:02.764322 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:02.764341 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:02.764567 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:02.765200 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:02.765223 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:02.765237 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:02.765669 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:02.765872 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:02.766914 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0502 14:11:02.767289 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":1220,"threads_started":1}
I20240502 14:11:02.769222 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.738197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41141 (local address 127.0.0.1:44568)
0502 14:11:02.741038 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.741101 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:02.746379 (+ 5278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:02.747461 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:02.747505 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:02.748739 (+ 1234us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:02.750193 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:02.750272 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.751640 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:02.751676 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:02.752600 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:02.752634 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.752908 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:02.753723 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:02.753787 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:02.756871 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:02.763439 (+ 6568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:02.763461 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:02.763486 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:02.764060 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:02.764758 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:02.764778 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:02.764793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:02.764996 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:02.765886 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:02.765907 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:02.766348 (+ 441us) client_negotiation.cc:769] Sending connection context
0502 14:11:02.766683 (+ 335us) client_negotiation.cc:240] Negotiation successful
0502 14:11:02.767136 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2174,"thread_start_us":1726,"threads_started":1}
I20240502 14:11:02.771170 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
Entry for principal kudu/127.27.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab.
I20240502 14:11:02.775017 28345 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
/tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:45501
--webserver_interface=127.27.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.174.84:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 02 14:11:02 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:11:02.794268 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.772762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.27.174.125:34987)
0502 14:11:02.775340 (+ 2578us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:02.775375 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:02.775436 (+ 61us) server_negotiation.cc:407] Connection header received
0502 14:11:02.775558 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:02.775577 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:02.775805 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:02.775955 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:02.777193 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.778691 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:02.779727 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.780229 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:02.784377 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:02.784470 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:02.784527 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:02.784621 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:02.788317 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:02.789552 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:02.789573 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:02.789592 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:02.789747 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:02.790436 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:02.790471 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:02.790483 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:02.790920 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:02.791127 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:02.792049 (+ 922us) server_negotiation.cc:299] Negotiation successful
0502 14:11:02.792634 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":2125,"thread_start_us":1417,"threads_started":1}
I20240502 14:11:02.795234 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.772478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59254)
0502 14:11:02.773022 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.773050 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:02.773196 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:02.776014 (+ 2818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:02.776047 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:02.776623 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:02.776950 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:02.776977 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.778890 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:02.778909 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:02.779523 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:02.779551 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:02.779804 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:02.780600 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:02.780652 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:02.784127 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:02.788528 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:02.788564 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:02.788596 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:02.789359 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:02.789940 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:02.789978 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:02.790008 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:02.790266 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:02.791150 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:02.791187 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:02.791507 (+ 320us) client_negotiation.cc:769] Sending connection context
0502 14:11:02.791884 (+ 377us) client_negotiation.cc:240] Negotiation successful
0502 14:11:02.792417 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20240502 14:11:02.797983 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:11:02.800467 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.799143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51482)
0502 14:11:02.799765 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.799977 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20240502 14:11:02.800922 13862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:02.805258 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.803864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51496)
0502 14:11:02.804508 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.804720 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20240502 14:11:02.805697 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:02.855172 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 1
I20240502 14:11:02.857729 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.856222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51500)
0502 14:11:02.856934 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.857143 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20240502 14:11:02.860733 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.859037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51516)
0502 14:11:02.859901 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.860115 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20240502 14:11:02.861197 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:02.929647 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 2
I20240502 14:11:02.932058 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.930925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51532)
0502 14:11:02.931461 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.931618 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20240502 14:11:02.934178 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:02.933207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51536)
0502 14:11:02.933677 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:02.933806 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20240502 14:11:02.934535 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.025894 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 3
I20240502 14:11:03.028074 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.027038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51548)
0502 14:11:03.027523 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.027670 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20240502 14:11:03.030320 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.029245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51558)
0502 14:11:03.029778 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.029911 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20240502 14:11:03.030680 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
W20240502 14:11:03.084419 13920 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240502 14:11:03.085238 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:11:03.085582 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:11:03.086208 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:11:03.115123 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:11:03.115463 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240502 14:11:03.115758 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:11:03.115980 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:11:03.116209 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:11:03.149966 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.174.84:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.174.126:41141,127.27.174.125:34987,127.27.174.124:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/krb5kdc/kudu_127.27.174.124.keytab
--principal=kudu/127.27.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.174.124:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on 1a15035a6830
build id 92
TSAN enabled
I20240502 14:11:03.151177 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:11:03.152166 13920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:11:03.167160 13927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:03.174096 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 4
I20240502 14:11:03.176244 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.175207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51568)
0502 14:11:03.175745 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.175890 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20240502 14:11:03.178303 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.177367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51572)
0502 14:11:03.177798 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.177923 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20240502 14:11:03.178619 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.423157 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } attempt: 7
May 02 14:11:03 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:11:03.452771 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.424663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.27.174.125:34987)
0502 14:11:03.426896 (+ 2233us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:03.426924 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:03.427371 (+ 447us) server_negotiation.cc:407] Connection header received
0502 14:11:03.427705 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:03.427723 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:03.427880 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:03.428029 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:03.429464 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:03.430308 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:03.431427 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:03.432301 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:03.442598 (+ 10297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:03.442675 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:03.442695 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:03.442790 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:03.446613 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:03.448426 (+ 1813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:03.448463 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:03.448476 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:03.448621 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:03.449464 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:03.449482 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:03.449495 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:03.449866 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:03.450030 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:03.451151 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0502 14:11:03.451452 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1635,"thread_start_us":1373,"threads_started":1}
I20240502 14:11:03.455062 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.424693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59258)
0502 14:11:03.427207 (+ 2514us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.427242 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:03.427492 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:03.428124 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:03.428151 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:03.428785 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:03.429198 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:03.429249 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:03.430541 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:03.430562 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:03.431199 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:03.431233 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:03.431503 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:03.438139 (+ 6636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:03.438202 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:03.442246 (+ 4044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:03.446888 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:03.446938 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:03.446977 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:03.448131 (+ 1154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:03.448820 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:03.448865 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:03.448913 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:03.449233 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:03.450075 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:03.450123 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:03.450528 (+ 405us) client_negotiation.cc:769] Sending connection context
0502 14:11:03.451011 (+ 483us) client_negotiation.cc:240] Negotiation successful
0502 14:11:03.451741 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"thread_start_us":1702,"threads_started":1}
I20240502 14:11:03.457242 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:11:03.460337 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.458328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51588)
0502 14:11:03.459251 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.459552 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20240502 14:11:03.465132 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.463014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51596)
0502 14:11:03.463962 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.464247 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20240502 14:11:03.465634 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.478013 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 5
I20240502 14:11:03.480351 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.479341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51608)
0502 14:11:03.479854 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.479992 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20240502 14:11:03.482692 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.481632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51614)
0502 14:11:03.482191 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.482317 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20240502 14:11:03.483115 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.524164 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 1
I20240502 14:11:03.527540 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.525242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51626)
0502 14:11:03.526356 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.526700 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20240502 14:11:03.530854 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.528819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51632)
0502 14:11:03.529770 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.530047 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20240502 14:11:03.531343 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.612810 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 2
I20240502 14:11:03.615538 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.614383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51644)
0502 14:11:03.614961 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.615121 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20240502 14:11:03.617722 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.616718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51652)
0502 14:11:03.617210 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.617345 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20240502 14:11:03.618136 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.703750 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 3
I20240502 14:11:03.706488 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.705356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51666)
0502 14:11:03.705931 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.706113 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20240502 14:11:03.708809 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.707727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51682)
0502 14:11:03.708235 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.708376 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20240502 14:11:03.709404 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:03.849928 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 4
I20240502 14:11:03.852520 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.851403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51686)
0502 14:11:03.851958 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.852114 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20240502 14:11:03.854735 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:03.853657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51696)
0502 14:11:03.854187 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:03.854330 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20240502 14:11:03.855123 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:04.030532 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 6
I20240502 14:11:04.034530 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.031972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51702)
0502 14:11:04.033850 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.033991 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":1140,"threads_started":1}
I20240502 14:11:04.038183 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.037133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51714)
0502 14:11:04.037626 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.037777 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20240502 14:11:04.038568 13913 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
I20240502 14:11:04.138500 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 5
I20240502 14:11:04.140683 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.139587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51722)
0502 14:11:04.140130 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.140292 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20240502 14:11:04.143029 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.141813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51726)
0502 14:11:04.142360 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.142496 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20240502 14:11:04.143456 13835 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.174.124:45501: Network error: Client connection negotiation failed: client connection to 127.27.174.124:45501: connect: Connection refused (error 111)
W20240502 14:11:03.167201 13928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:03.169481 13920 server_base.cc:1034] running on GCE node
W20240502 14:11:03.167438 13930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:11:04.238534 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:11:04.241021 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:11:04.242359 13920 hybrid_clock.cc:648] HybridClock initialized: now 1714659064242314 us; error 64 us; skew 500 ppm
I20240502 14:11:04.243137 13920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240502 14:11:04.248608 13920 init.cc:377] Logged in from keytab as kudu/127.27.174.124@KRBTEST.COM (short username kudu)
I20240502 14:11:04.252656 13920 webserver.cc:460] Webserver started at http://127.27.174.124:40985/ using document root <none> and password file <none>
I20240502 14:11:04.253489 13920 fs_manager.cc:359] Metadata directory not provided
I20240502 14:11:04.253690 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:11:04.254117 13920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:11:04.259380 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data/instance:
uuid: "fb7ad1f9dfac419ca165a0e6fa4126af"
format_stamp: "Formatted at 2024-05-02 14:11:04 on dist-test-slave-dlb8"
server_key: "cf9edde873fca434f0f9967932a25ee4"
server_key_iv: "fc9fb5528bef467ab6a96376d704de5f"
server_key_version: "encryptionkey@0"
I20240502 14:11:04.260457 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal/instance:
uuid: "fb7ad1f9dfac419ca165a0e6fa4126af"
format_stamp: "Formatted at 2024-05-02 14:11:04 on dist-test-slave-dlb8"
server_key: "cf9edde873fca434f0f9967932a25ee4"
server_key_iv: "fc9fb5528bef467ab6a96376d704de5f"
server_key_version: "encryptionkey@0"
I20240502 14:11:04.267220 13920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 14:11:04.272223 13943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:11:04.273103 13920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:11:04.273420 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data,/tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal
uuid: "fb7ad1f9dfac419ca165a0e6fa4126af"
format_stamp: "Formatted at 2024-05-02 14:11:04 on dist-test-slave-dlb8"
server_key: "cf9edde873fca434f0f9967932a25ee4"
server_key_iv: "fc9fb5528bef467ab6a96376d704de5f"
server_key_version: "encryptionkey@0"
I20240502 14:11:04.273761 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:11:04.329046 13920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:11:04.334074 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:11:04.334524 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:11:04.401260 13920 rpc_server.cc:307] RPC server started. Bound to: 127.27.174.124:45501
I20240502 14:11:04.401325 13994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.174.124:45501 every 8 connection(s)
I20240502 14:11:04.404179 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/data/info.pb
I20240502 14:11:04.409116 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:11:04.411126 28345 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu as pid 13920
I20240502 14:11:04.411760 28345 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTLmxo8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714658657003751-28345-0/minicluster-data/master-2/wal/instance
I20240502 14:11:04.413247 28345 external_mini_cluster.cc:1428] Setting key e5b4f7c259d68e1edad3bc53188874ce
I20240502 14:11:04.430243 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
I20240502 14:11:04.466339 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.416444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.27.174.124:45501)
0502 14:11:04.419307 (+ 2863us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.419341 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.419408 (+ 67us) server_negotiation.cc:407] Connection header received
0502 14:11:04.424956 (+ 5548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.425013 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.425533 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.426168 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.433643 (+ 7475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.435820 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.441647 (+ 5827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.442409 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.454216 (+ 11807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.454338 (+ 122us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.454378 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.454514 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.459475 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.460812 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.460852 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.460900 (+ 48us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.461092 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.461688 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.461705 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.461722 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.462363 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.462681 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.463535 (+ 854us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.464096 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020,"thread_start_us":1879,"threads_started":1}
I20240502 14:11:04.474571 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.435190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.27.174.126:41141)
0502 14:11:04.436959 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.436976 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.454134 (+ 17158us) server_negotiation.cc:407] Connection header received
0502 14:11:04.454301 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.454326 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.454542 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.454751 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.457565 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.458762 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.460131 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.460458 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.464929 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.465045 (+ 116us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.465066 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.465159 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.468815 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.469753 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.469772 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.469785 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.469911 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.470514 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.470549 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.470564 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.470962 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.471194 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.472038 (+ 844us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.472456 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404,"thread_start_us":1145,"threads_started":1}
I20240502 14:11:04.474601 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.435147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.126:41141 (local address 127.0.0.1:44574)
0502 14:11:04.453432 (+ 18285us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.453556 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:04.453840 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:04.454867 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:04.454908 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:04.456385 (+ 1477us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:04.457241 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.457298 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.458996 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.459033 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:04.459899 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.459947 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.460276 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.461042 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:04.461109 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:04.464681 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:04.468982 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.469006 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.469024 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.469576 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.470107 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.470124 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.470137 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.470290 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.471210 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:04.471258 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:04.471544 (+ 286us) client_negotiation.cc:769] Sending connection context
0502 14:11:04.471819 (+ 275us) client_negotiation.cc:240] Negotiation successful
0502 14:11:04.472254 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17510,"thread_start_us":16655,"threads_started":1}
I20240502 14:11:04.476742 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.125@KRBTEST.COM
I20240502 14:11:04.492161 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.469906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.27.174.125:34987)
0502 14:11:04.471918 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.471934 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.471965 (+ 31us) server_negotiation.cc:407] Connection header received
0502 14:11:04.472062 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.472076 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.472217 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.472359 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.474108 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.475062 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.476335 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.476801 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.481394 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.481546 (+ 152us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.481592 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.481774 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.485618 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.486616 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.486646 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.486666 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.486846 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.488615 (+ 1769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.488633 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.488649 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.488996 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.489296 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.489837 (+ 541us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.490184 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1637,"thread_start_us":1391,"threads_started":1}
I20240502 14:11:04.499567 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.478102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.125:34987 (local address 127.0.0.1:59262)
0502 14:11:04.478982 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.479038 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:04.479232 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:04.483547 (+ 4315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:04.483572 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:04.484105 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:04.484433 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.484462 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.485882 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.485901 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:04.486565 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.486627 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.486896 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.487486 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:04.487542 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:04.489883 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:04.494569 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.494596 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.494625 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.495197 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.495729 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.495748 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.495765 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.495925 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.496743 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:04.496760 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:04.496940 (+ 180us) client_negotiation.cc:769] Sending connection context
0502 14:11:04.497147 (+ 207us) client_negotiation.cc:240] Negotiation successful
0502 14:11:04.497538 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20240502 14:11:04.499879 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.478020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.27.174.125:34987)
0502 14:11:04.482933 (+ 4913us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.482960 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.483014 (+ 54us) server_negotiation.cc:407] Connection header received
0502 14:11:04.483145 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.483161 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.483313 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.483457 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.484688 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.485650 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.486803 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.487237 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.490140 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.490218 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.490251 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.490360 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.494325 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.495390 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.495412 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.495429 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.495561 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.496080 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.496098 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.496110 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.496444 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.496640 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.497356 (+ 716us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.497731 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4403,"thread_start_us":4130,"threads_started":1}
I20240502 14:11:04.501536 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.174.126@KRBTEST.COM
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.124@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20240502 14:11:04.514691 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.494635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.27.174.126:41141)
0502 14:11:04.495494 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.495532 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.495578 (+ 46us) server_negotiation.cc:407] Connection header received
0502 14:11:04.495716 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.495734 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.495915 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.496040 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.498127 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.498807 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.499867 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.500707 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.505667 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.505738 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.505759 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.505873 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.509541 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.510701 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.510729 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.510749 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.510920 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.511472 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.511503 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.511531 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.511958 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.512246 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.512861 (+ 615us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.513126 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20240502 14:11:04.521178 28345 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240502 14:11:04.521541 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.502659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51736)
0502 14:11:04.503435 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.503474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:04.503666 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:04.504428 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:04.504460 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:04.505246 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:04.505715 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.505755 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.507180 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.507199 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:04.507777 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.507808 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.508078 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.508769 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:04.508825 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:04.511556 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:04.516133 (+ 4577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.516160 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.516180 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.516893 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.517409 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.517428 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.517442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.517603 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.518448 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:04.518476 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:04.518674 (+ 198us) client_negotiation.cc:769] Sending connection context
0502 14:11:04.518880 (+ 206us) client_negotiation.cc:240] Negotiation successful
0502 14:11:04.519275 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20240502 14:11:04.521673 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.502933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.27.174.124:45501)
0502 14:11:04.503684 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.503703 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.503734 (+ 31us) server_negotiation.cc:407] Connection header received
0502 14:11:04.503829 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.503844 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.503983 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.504104 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.505967 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.506990 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.507998 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.508495 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.511782 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.511843 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.511858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.511944 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.515949 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.517066 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.517089 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.517102 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.517237 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.517776 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.517804 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.517825 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.518276 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.518476 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.519071 (+ 595us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.519550 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":92}
I20240502 14:11:04.531364 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: Bootstrap starting.
I20240502 14:11:04.536172 13995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: Neither blocks nor log segments found. Creating new log.
I20240502 14:11:04.537953 13995 log.cc:824] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: Log is configured to *not* fsync() on all Append() calls
I20240502 14:11:04.531639 13810 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6148c0bc14fd40f6a81369ca9948cff7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584
I20240502 14:11:04.542800 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: No bootstrap required, opened a new log
I20240502 14:11:04.560973 13995 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.561599 13995 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7ad1f9dfac419ca165a0e6fa4126af, State: Initialized, Role: FOLLOWER
I20240502 14:11:04.562184 13995 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.564409 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.565116 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:04.566188 13995 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: configured and running, proceeding with master startup.
W20240502 14:11:04.589459 13949 tablet.cc:2369] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:11:04.590353 14015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:11:04.590626 14015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:11:04.586570 13920 master_runner.cc:186] Error getting master registration for 127.27.174.126:41141: OK, instance_id { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" instance_seqno: 1714659061035274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 14:11:04.618377 13920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 14:11:04.695884 13835 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 6
May 02 14:11:04 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.126@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
I20240502 14:11:04.725447 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.697171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51750)
0502 14:11:04.702774 (+ 5603us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:04.702822 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:04.703192 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:04.703928 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:04.703957 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:04.704913 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:04.705544 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.705603 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.707269 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.707305 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:04.708362 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:04.708417 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.708730 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.710933 (+ 2203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:04.710987 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:04.713994 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:04.719602 (+ 5608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.719628 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.719644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.720124 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.720770 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:04.720794 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:04.720814 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:04.721016 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:04.722029 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:04.722076 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:04.722343 (+ 267us) client_negotiation.cc:769] Sending connection context
0502 14:11:04.722597 (+ 254us) client_negotiation.cc:240] Negotiation successful
0502 14:11:04.723077 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4983,"thread_start_us":4675,"threads_started":1}
I20240502 14:11:04.725423 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.697374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.27.174.124:45501)
0502 14:11:04.698518 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.698547 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.702954 (+ 4407us) server_negotiation.cc:407] Connection header received
0502 14:11:04.703427 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.703452 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.703674 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.703841 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.705825 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.707036 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.708613 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.709118 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.714323 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.714392 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.714411 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.714500 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.719390 (+ 4890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.720307 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.720339 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.720364 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.720552 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.721232 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.721259 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.721278 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.721780 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.722055 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.722801 (+ 746us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.723201 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20240502 14:11:04.736064 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7: Bootstrap starting.
I20240502 14:11:04.743059 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7: Neither blocks nor log segments found. Creating new log.
I20240502 14:11:04.745348 13835 log.cc:824] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7: Log is configured to *not* fsync() on all Append() calls
I20240502 14:11:04.750982 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7: No bootstrap required, opened a new log
I20240502 14:11:04.769970 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.770815 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6148c0bc14fd40f6a81369ca9948cff7, State: Initialized, Role: FOLLOWER
I20240502 14:11:04.771663 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.774058 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.774802 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:04.775862 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:11:04.775967 13810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:11:04.776310 13810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:11:04.781953 13810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.785976 13970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6148c0bc14fd40f6a81369ca9948cff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af"
I20240502 14:11:04.786679 13970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:11:04.792958 13970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6148c0bc14fd40f6a81369ca9948cff7 in term 1.
I20240502 14:11:04.794353 13784 leader_election.cc:302] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6148c0bc14fd40f6a81369ca9948cff7, fb7ad1f9dfac419ca165a0e6fa4126af; no voters:
I20240502 14:11:04.795415 14019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:11:04.796041 13810 leader_election.cc:288] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [CANDIDATE]: Term 1 election: Requested vote from peers 245151308cf44e509d776584225ff968 (127.27.174.125:34987), fb7ad1f9dfac419ca165a0e6fa4126af (127.27.174.124:45501)
I20240502 14:11:04.797681 14019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [term 1 LEADER]: Becoming Leader. State: Replica: 6148c0bc14fd40f6a81369ca9948cff7, State: Running, Role: LEADER
I20240502 14:11:04.798655 14019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:04.809669 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6148c0bc14fd40f6a81369ca9948cff7. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.810279 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:11:04.813194 14033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:11:04.831593 14033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:11:04.879387 13970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [term 1 FOLLOWER]: Refusing update from remote peer 6148c0bc14fd40f6a81369ca9948cff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:11:04.880952 14028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:11:04.885141 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.826432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.27.174.125:34987)
0502 14:11:04.827588 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.827625 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.836754 (+ 9129us) server_negotiation.cc:407] Connection header received
0502 14:11:04.837711 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.837737 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.837943 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.838099 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.853071 (+ 14972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.854251 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.857164 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.857669 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.865974 (+ 8305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.866077 (+ 103us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.866108 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.866242 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.873505 (+ 7263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.876927 (+ 3422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.876979 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.877000 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.877307 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.878549 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.878576 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.878618 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.879307 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.879773 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.881221 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.881888 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20240502 14:11:04.896935 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.846664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.27.174.124:45501)
0502 14:11:04.847633 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.847661 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.867390 (+ 19729us) server_negotiation.cc:407] Connection header received
0502 14:11:04.867726 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.867752 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.867993 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.868166 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.870835 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.871939 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.874368 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.874884 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.878434 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.878521 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.878548 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.878660 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.883130 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.884521 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.884551 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.884570 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.884782 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.885525 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.885560 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.885588 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.886165 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.886503 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.891850 (+ 5347us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.892629 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20240502 14:11:04.923461 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6148c0bc14fd40f6a81369ca9948cff7. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.924199 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:04.924362 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:04.827995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.27.174.126:41141)
0502 14:11:04.828929 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:04.828962 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:04.851895 (+ 22933us) server_negotiation.cc:407] Connection header received
0502 14:11:04.852937 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:04.852978 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:04.853286 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:04.853467 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:04.885974 (+ 32507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.887161 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:04.892829 (+ 5668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:04.893357 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:04.910645 (+ 17288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:04.910750 (+ 105us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:04.910781 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:04.910925 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:04.915644 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.917096 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.917134 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.917157 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.917354 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:04.918121 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:04.918152 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:04.918178 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:04.918719 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:04.919047 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:04.921098 (+ 2051us) server_negotiation.cc:299] Negotiation successful
0502 14:11:04.921530 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20240502 14:11:04.862960 13766 master_runner.cc:186] Error getting master registration for 127.27.174.125:34987: OK, instance_id { permanent_uuid: "245151308cf44e509d776584225ff968" instance_seqno: 1714659062693002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 14:11:04.931221 13766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 14:11:04.934810 14042 mvcc.cc:205] Tried to move back new op lower bound from 7023243529729687552 to 7023243529424805888. Current Snapshot: MvccSnapshot[applied={T|T < 7023243529729687552}]
I20240502 14:11:04.937801 14043 mvcc.cc:205] Tried to move back new op lower bound from 7023243529729687552 to 7023243529424805888. Current Snapshot: MvccSnapshot[applied={T|T < 7023243529729687552}]
I20240502 14:11:04.939880 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.940567 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:04.947104 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:04.947913 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: This master's current role is: LEADER
I20240502 14:11:04.962735 14033 catalog_manager.cc:1347] Generated new cluster ID: f4f8128d7cca4684b2f6cdf5563e0570
I20240502 14:11:04.963023 14033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:11:04.997630 14033 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:11:05.000454 14033 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:11:05.034269 14033 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7: Generated new TSK 0
I20240502 14:11:05.035401 14033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:11:05.099016 13913 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } attempt: 7
W20240502 14:11:05.155272 13999 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.145986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.27.174.126:41141)
0502 14:11:05.147512 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.147537 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.149512 (+ 1975us) server_negotiation.cc:407] Connection header received
0502 14:11:05.150528 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.150555 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.150768 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.150897 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.154286 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44602: BlockingRecv error: recv got EOF from 127.0.0.1:44602 (error 108)
Metrics: {"server-negotiator.queue_time_us":948}
May 02 14:11:05 dist-test-slave-dlb8 krb5kdc[13753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714659062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.174.125@KRBTEST.COM for kudu/127.27.174.124@KRBTEST.COM
W20240502 14:11:05.162359 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.148791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.27.174.125:34987)
0502 14:11:05.149393 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.149412 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.157905 (+ 8493us) server_negotiation.cc:407] Connection header received
0502 14:11:05.158547 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.158572 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.158801 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.158997 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.161125 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59292: BlockingRecv error: recv got EOF from 127.0.0.1:59292 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20240502 14:11:05.186885 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:05.100297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.174.124:45501 (local address 127.0.0.1:51760)
0502 14:11:05.133782 (+ 33485us) negotiation.cc:107] Waiting for socket to connect
0502 14:11:05.133930 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0502 14:11:05.134876 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:11:05.136860 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 14:11:05.136923 (+ 63us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 14:11:05.146486 (+ 9563us) client_negotiation.cc:189] Negotiated authn=SASL
0502 14:11:05.147098 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:05.147154 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:05.149635 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 14:11:05.149673 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 14:11:05.150968 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 14:11:05.151037 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 14:11:05.152386 (+ 1349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:05.153191 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0502 14:11:05.153268 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0502 14:11:05.157523 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 14:11:05.171230 (+ 13707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:05.171273 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:05.171307 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:05.172255 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:05.182092 (+ 9837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0502 14:11:05.182131 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0502 14:11:05.182155 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0502 14:11:05.182409 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0502 14:11:05.183515 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0502 14:11:05.183547 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0502 14:11:05.183819 (+ 272us) client_negotiation.cc:769] Sending connection context
0502 14:11:05.184121 (+ 302us) client_negotiation.cc:240] Negotiation successful
0502 14:11:05.184604 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31577,"thread_start_us":30165,"threads_started":1}
W20240502 14:11:05.188033 14058 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.153145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.27.174.124:45501)
0502 14:11:05.174542 (+ 21397us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.174569 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.174627 (+ 58us) server_negotiation.cc:407] Connection header received
0502 14:11:05.174791 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.174818 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.175101 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.175292 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.184559 (+ 9267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51772: BlockingRecv error: recv got EOF from 127.0.0.1:51772 (error 108)
Metrics: {"server-negotiator.queue_time_us":20667,"thread_start_us":20670,"threads_started":1}
I20240502 14:11:05.191504 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:05.101632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.27.174.124:45501)
0502 14:11:05.102410 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.102436 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.134225 (+ 31789us) server_negotiation.cc:407] Connection header received
0502 14:11:05.135444 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.135484 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.135809 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.136014 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.147434 (+ 11420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:05.148571 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:05.151306 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:05.151879 (+ 573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:05.163575 (+ 11696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0502 14:11:05.163690 (+ 115us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0502 14:11:05.163730 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0502 14:11:05.163896 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0502 14:11:05.170895 (+ 6999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:05.176691 (+ 5796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:05.176758 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:05.177845 (+ 1087us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:05.178235 (+ 390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0502 14:11:05.182673 (+ 4438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0502 14:11:05.182704 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0502 14:11:05.182726 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0502 14:11:05.183245 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0502 14:11:05.183536 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:05.187797 (+ 4261us) server_negotiation.cc:299] Negotiation successful
0502 14:11:05.188591 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20240502 14:11:05.203138 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: Bootstrap starting.
I20240502 14:11:05.213637 13913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: Neither blocks nor log segments found. Creating new log.
I20240502 14:11:05.218560 13913 log.cc:824] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: Log is configured to *not* fsync() on all Append() calls
W20240502 14:11:05.227339 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.219625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.27.174.125:34987)
0502 14:11:05.220251 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.220274 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.223602 (+ 3328us) server_negotiation.cc:407] Connection header received
0502 14:11:05.224154 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.224183 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.224457 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.224586 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.226433 (+ 1847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59302: BlockingRecv error: recv got EOF from 127.0.0.1:59302 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20240502 14:11:05.243877 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: No bootstrap required, opened a new log
W20240502 14:11:05.285385 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.270429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.27.174.125:34987)
0502 14:11:05.271101 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.271125 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.279669 (+ 8544us) server_negotiation.cc:407] Connection header received
0502 14:11:05.280606 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.280632 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.280884 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.281012 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.284400 (+ 3388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59318: BlockingRecv error: recv got EOF from 127.0.0.1:59318 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20240502 14:11:05.287099 13913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:05.287940 13913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245151308cf44e509d776584225ff968, State: Initialized, Role: FOLLOWER
I20240502 14:11:05.288986 13913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } }
I20240502 14:11:05.296792 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:05.297538 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:05.300034 13913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:11:05.300199 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6148c0bc14fd40f6a81369ca9948cff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "245151308cf44e509d776584225ff968"
I20240502 14:11:05.300859 13888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:11:05.308507 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6148c0bc14fd40f6a81369ca9948cff7 in term 1.
I20240502 14:11:05.309844 13887 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [term 1 FOLLOWER]: Refusing update from remote peer 6148c0bc14fd40f6a81369ca9948cff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:11:05.316833 14047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:11:05.366199 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.344484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.27.174.125:34987)
0502 14:11:05.345167 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.345192 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.361155 (+ 15963us) server_negotiation.cc:407] Connection header received
0502 14:11:05.362990 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.363016 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.363284 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.363415 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.365177 (+ 1762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59320: BlockingRecv error: recv got EOF from 127.0.0.1:59320 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20240502 14:11:05.404565 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6148c0bc14fd40f6a81369ca9948cff7. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:05.410167 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:11:05.404932 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:05.410733 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 14:11:05.424793 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.411494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.27.174.125:34987)
0502 14:11:05.412144 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.412169 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.420570 (+ 8401us) server_negotiation.cc:407] Connection header received
0502 14:11:05.421474 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.421516 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.421796 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.421984 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.423807 (+ 1823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59332: BlockingRecv error: recv got EOF from 127.0.0.1:59332 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20240502 14:11:05.425761 14097 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20240502 14:11:05.483825 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.470633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.27.174.125:34987)
0502 14:11:05.471302 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.471322 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.479687 (+ 8365us) server_negotiation.cc:407] Connection header received
0502 14:11:05.480661 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.480695 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.480976 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.481123 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.482914 (+ 1791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59340: BlockingRecv error: recv got EOF from 127.0.0.1:59340 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20240502 14:11:05.519941 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.514478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.27.174.125:34987)
0502 14:11:05.515180 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.515206 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.516588 (+ 1382us) server_negotiation.cc:407] Connection header received
0502 14:11:05.517014 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.517041 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.517328 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.517504 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.518999 (+ 1495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59346: BlockingRecv error: recv got EOF from 127.0.0.1:59346 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20240502 14:11:05.565151 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.559449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.27.174.125:34987)
0502 14:11:05.560151 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.560179 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.561558 (+ 1379us) server_negotiation.cc:407] Connection header received
0502 14:11:05.562091 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.562120 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.562461 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.562646 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.564114 (+ 1468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59358: BlockingRecv error: recv got EOF from 127.0.0.1:59358 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20240502 14:11:05.596088 14015 catalog_manager.cc:1259] Loaded cluster ID: f4f8128d7cca4684b2f6cdf5563e0570
I20240502 14:11:05.596379 14015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: loading cluster ID for follower catalog manager: success
I20240502 14:11:05.602937 14015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: acquiring CA information for follower catalog manager: success
I20240502 14:11:05.607831 14015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb7ad1f9dfac419ca165a0e6fa4126af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240502 14:11:05.647217 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.641601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.27.174.125:34987)
0502 14:11:05.642334 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.642364 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.643752 (+ 1388us) server_negotiation.cc:407] Connection header received
0502 14:11:05.644205 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.644233 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.644531 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.644724 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.646237 (+ 1513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59368: BlockingRecv error: recv got EOF from 127.0.0.1:59368 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20240502 14:11:05.794281 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:05.788860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.27.174.125:34987)
0502 14:11:05.789493 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:05.789518 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:05.791054 (+ 1536us) server_negotiation.cc:407] Connection header received
0502 14:11:05.791472 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:05.791498 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:05.791798 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:05.791989 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:05.793371 (+ 1382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59382: BlockingRecv error: recv got EOF from 127.0.0.1:59382 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20240502 14:11:05.821599 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6148c0bc14fd40f6a81369ca9948cff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6148c0bc14fd40f6a81369ca9948cff7" member_type: VOTER last_known_addr { host: "127.27.174.126" port: 41141 } } peers { permanent_uuid: "245151308cf44e509d776584225ff968" member_type: VOTER last_known_addr { host: "127.27.174.125" port: 34987 } } peers { permanent_uuid: "fb7ad1f9dfac419ca165a0e6fa4126af" member_type: VOTER last_known_addr { host: "127.27.174.124" port: 45501 } } }
I20240502 14:11:05.822358 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6148c0bc14fd40f6a81369ca9948cff7 [sys.catalog]: This master's current role is: LEADER
W20240502 14:11:06.097883 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:11:06.092281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.27.174.125:34987)
0502 14:11:06.092915 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.092935 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.094181 (+ 1246us) server_negotiation.cc:407] Connection header received
0502 14:11:06.094626 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.094655 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.094954 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.095102 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0502 14:11:06.096924 (+ 1822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59398: BlockingRecv error: recv got EOF from 127.0.0.1:59398 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20240502 14:11:06.432968 14097 catalog_manager.cc:1259] Loaded cluster ID: f4f8128d7cca4684b2f6cdf5563e0570
I20240502 14:11:06.433334 14097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: loading cluster ID for follower catalog manager: success
I20240502 14:11:06.440260 14097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: acquiring CA information for follower catalog manager: success
I20240502 14:11:06.445436 14097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 245151308cf44e509d776584225ff968: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:11:06.682133 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:06.649104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.27.174.125:34987)
0502 14:11:06.670727 (+ 21623us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.670752 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.670801 (+ 49us) server_negotiation.cc:407] Connection header received
0502 14:11:06.670955 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.670981 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.671315 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.671491 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:11:06.673124 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.674983 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:06.677739 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.678312 (+ 573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:06.678579 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:11:06.679257 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:11:06.679427 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:06.680447 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0502 14:11:06.680605 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19972,"thread_start_us":21174,"threads_started":1}
I20240502 14:11:06.779650 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:06.746607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.27.174.124:45501)
0502 14:11:06.765989 (+ 19382us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.766017 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.766097 (+ 80us) server_negotiation.cc:407] Connection header received
0502 14:11:06.766318 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.766344 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.766724 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.766965 (+ 241us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:11:06.768566 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.773526 (+ 4960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:06.775608 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.776142 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:06.776365 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:11:06.776998 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:11:06.777176 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:06.778141 (+ 965us) server_negotiation.cc:299] Negotiation successful
0502 14:11:06.778297 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18545,"thread_start_us":18207,"threads_started":1}
I20240502 14:11:06.872514 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:06.820871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.27.174.126:41141)
0502 14:11:06.850963 (+ 30092us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.850989 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.851034 (+ 45us) server_negotiation.cc:407] Connection header received
0502 14:11:06.851231 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.851255 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.851546 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.851725 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:11:06.854547 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.856161 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:06.866675 (+ 10514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.867191 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:06.867417 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:11:06.867918 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:11:06.868080 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:06.870920 (+ 2840us) server_negotiation.cc:299] Negotiation successful
0502 14:11:06.871080 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29527,"thread_start_us":30990,"threads_started":1}
I20240502 14:11:06.881709 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13766
I20240502 14:11:06.885922 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:06.840547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.27.174.125:34987)
0502 14:11:06.841184 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.841215 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.853388 (+ 12173us) server_negotiation.cc:407] Connection header received
0502 14:11:06.856842 (+ 3454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.856869 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.857146 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.857265 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:11:06.863286 (+ 6021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.864446 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:06.873067 (+ 8621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.873560 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:06.883491 (+ 9931us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:11:06.883759 (+ 268us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:11:06.883865 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:06.884299 (+ 434us) server_negotiation.cc:299] Negotiation successful
0502 14:11:06.884462 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20240502 14:11:06.892657 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0502 14:11:06.861649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.27.174.124:45501)
0502 14:11:06.862388 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0502 14:11:06.862411 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0502 14:11:06.869922 (+ 7511us) server_negotiation.cc:407] Connection header received
0502 14:11:06.883662 (+ 13740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0502 14:11:06.883688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0502 14:11:06.883954 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0502 14:11:06.884058 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0502 14:11:06.885320 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.886616 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0502 14:11:06.889304 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0502 14:11:06.889816 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 14:11:06.890432 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0502 14:11:06.890691 (+ 259us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0502 14:11:06.890796 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0502 14:11:06.891168 (+ 372us) server_negotiation.cc:299] Negotiation successful
0502 14:11:06.891367 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20240502 14:11:06.931676 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13842
I20240502 14:11:06.957324 28345 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTLmxo8/build/tsan/bin/kudu with pid 13920
2024-05-02T14:11:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (8687 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (16218 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (409848 ms total)
[ PASSED ] 51 tests.
[ SKIPPED ] 5 tests, listed below:
[ SKIPPED ] SecurityITest.TestJwtMiniCluster
[ SKIPPED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
[ SKIPPED ] SecurityITest.IPKICACert
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedTSK
==================
WARNING: ThreadSanitizer: data race (pid=28345)
Write of size 8 at 0x7b0800003588 by main thread:
#0 operator delete(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 (security-itest+0x3b3a19)
#1 std::__1::_DeallocateCaller::__do_call(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3e08b9)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3e0859)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x126282)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126219)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139f09)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139e19)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139d01)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139c74)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x133059)
#10 cxa_at_exit_wrapper(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394 (security-itest+0x33379f)
Previous read of size 8 at 0x7b0800003588 by thread T20:
#0 std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>* std::__1::__tree_next_iter<std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*, std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:182:14 (libkrpc.so+0x13d9fa)
#1 std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>::operator++() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:929:11 (libkrpc.so+0x133e12)
#2 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1696:31 (libkrpc.so+0x13bb72)
#3 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b9c9)
#4 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133c30)
#5 kudu::rpc::ClientNegotiation::SendNegotiate() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b376)
#6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12abdb)
#7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x1809cb)
#8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17fb12)
#9 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19eeec)
#10 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ee69)
#11 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19edf9)
#12 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19edc1)
#13 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19e09d)
#14 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#15 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#16 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:775:7 (libkudu_util.so+0x455236)
#17 kudu::ThreadPool::CreateThread()::$_2::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:848:48 (libkudu_util.so+0x458691)
#18 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x458649)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x4585d9)
#20 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x4585a1)
#21 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45789d)
#22 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#23 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#24 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
As if synchronized via sleep:
#0 nanosleep /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363 (security-itest+0x358c82)
#1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5f1a)
#2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3cd336)
#3 kudu::Subprocess::KillAndWait(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x429cc0)
#4 kudu::clock::MiniChronyd::Stop() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150b0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14dae)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc025e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc01cd)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b7eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0ac9)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0aa0)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0a60)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb0a16)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb0934)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb072b)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96d11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8361e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83729)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e7fa7)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e7f0d)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e674b)
#21 kudu::SecurityITest::~SecurityITest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3de267)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x4133b5)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3de779)
#24 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x65467)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#27 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43dc2)
#28 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#29 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#32 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#33 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (libkudu_test_main.so+0x8c7b)
#34 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (libkudu_test_main.so+0x7a1c)
Thread T20 'client-negotiat' (tid=14150, finished) created by thread T23 at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (security-itest+0x337755)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43719a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3939)
#3 kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:847:10 (libkudu_util.so+0x45442b)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:642:21 (libkudu_util.so+0x45282f)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:524:10 (libkudu_util.so+0x454992)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198df8)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x199851)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199416)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4dae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x198782)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5fd3)
#12 ev_invoke_pending /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9982)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x197655)
#14 ev_run /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab2c)
#15 ev::loop_ref::run(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a2768)
#16 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19ac35)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d981)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d939)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d8c9)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d891)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19cb8d)
#22 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#23 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#24 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=28345)
Write of size 8 at 0x7b0800003590 by main thread:
#0 operator delete(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 (security-itest+0x3b3a19)
#1 std::__1::_DeallocateCaller::__do_call(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3e08b9)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3e0859)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x126282)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126219)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139f09)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139e19)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139d01)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139c74)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x133059)
#10 cxa_at_exit_wrapper(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394 (security-itest+0x33379f)
Previous read of size 8 at 0x7b0800003590 by thread T20:
#0 bool std::__1::__tree_is_left_child<std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:81:24 (libkrpc.so+0x13c53b)
#1 std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>* std::__1::__tree_next_iter<std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*, std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:184:13 (libkrpc.so+0x13da24)
#2 std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>::operator++() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:929:11 (libkrpc.so+0x133e12)
#3 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1696:31 (libkrpc.so+0x13bb72)
#4 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b9c9)
#5 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133c30)
#6 kudu::rpc::ClientNegotiation::SendNegotiate() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b376)
#7 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12abdb)
#8 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x1809cb)
#9 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17fb12)
#10 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19eeec)
#11 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ee69)
#12 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19edf9)
#13 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19edc1)
#14 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19e09d)
#15 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#16 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#17 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:775:7 (libkudu_util.so+0x455236)
#18 kudu::ThreadPool::CreateThread()::$_2::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:848:48 (libkudu_util.so+0x458691)
#19 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x458649)
#20 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x4585d9)
#21 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x4585a1)
#22 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45789d)
#23 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#24 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
As if synchronized via sleep:
#0 nanosleep /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363 (security-itest+0x358c82)
#1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5f1a)
#2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3cd336)
#3 kudu::Subprocess::KillAndWait(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x429cc0)
#4 kudu::clock::MiniChronyd::Stop() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150b0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14dae)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc025e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc01cd)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b7eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0ac9)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0aa0)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0a60)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb0a16)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb0934)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb072b)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96d11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8361e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83729)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e7fa7)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e7f0d)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e674b)
#21 kudu::SecurityITest::~SecurityITest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3de267)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x4133b5)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3de779)
#24 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x65467)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#27 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43dc2)
#28 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#29 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#32 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#33 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (libkudu_test_main.so+0x8c7b)
#34 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (libkudu_test_main.so+0x7a1c)
Thread T20 'client-negotiat' (tid=14150, finished) created by thread T23 at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (security-itest+0x337755)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43719a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3939)
#3 kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:847:10 (libkudu_util.so+0x45442b)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:642:21 (libkudu_util.so+0x45282f)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:524:10 (libkudu_util.so+0x454992)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198df8)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x199851)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199416)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4dae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x198782)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5fd3)
#12 ev_invoke_pending /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9982)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x197655)
#14 ev_run /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab2c)
#15 ev::loop_ref::run(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a2768)
#16 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19ac35)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d981)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d939)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d8c9)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d891)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19cb8d)
#22 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#23 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#24 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=28345)
Write of size 8 at 0x7b0800003598 by main thread:
#0 operator delete(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 (security-itest+0x3b3a19)
#1 std::__1::_DeallocateCaller::__do_call(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3e08b9)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3e0859)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x126282)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126219)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139f09)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139e19)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139d01)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139c74)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x133059)
#10 cxa_at_exit_wrapper(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394 (security-itest+0x33379f)
Previous read of size 4 at 0x7b080000359c by thread T20:
#0 void std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1809:35 (libkrpc.so+0x13cf4b)
#1 void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::__construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (libkrpc.so+0x13cf01)
#2 void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (libkrpc.so+0x13cc91)
#3 std::__1::unique_ptr<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>, std::__1::__tree_node_destructor<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > > > std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__construct_node<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:2194:5 (libkrpc.so+0x13c9e8)
#4 std::__1::__tree_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__emplace_multi<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:2242:25 (libkrpc.so+0x13c896)
#5 std::__1::__tree_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__insert_multi<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1326:16 (libkrpc.so+0x13bf08)
#6 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1697:9 (libkrpc.so+0x13bb6a)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b9c9)
#8 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133c30)
#9 kudu::rpc::ClientNegotiation::SendNegotiate() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b376)
#10 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12abdb)
#11 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x1809cb)
#12 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17fb12)
#13 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19eeec)
#14 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ee69)
#15 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19edf9)
#16 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19edc1)
#17 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19e09d)
#18 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#19 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#20 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:775:7 (libkudu_util.so+0x455236)
#21 kudu::ThreadPool::CreateThread()::$_2::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:848:48 (libkudu_util.so+0x458691)
#22 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x458649)
#23 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x4585d9)
#24 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x4585a1)
#25 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45789d)
#26 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#27 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#28 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
As if synchronized via sleep:
#0 nanosleep /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363 (security-itest+0x358c82)
#1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5f1a)
#2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3cd336)
#3 kudu::Subprocess::KillAndWait(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x429cc0)
#4 kudu::clock::MiniChronyd::Stop() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150b0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14dae)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc025e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc01cd)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b7eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0ac9)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0aa0)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0a60)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb0a16)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb0934)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb072b)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96d11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8361e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83729)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e7fa7)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e7f0d)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e674b)
#21 kudu::SecurityITest::~SecurityITest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3de267)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x4133b5)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3de779)
#24 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x65467)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#27 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43dc2)
#28 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#29 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#32 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#33 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (libkudu_test_main.so+0x8c7b)
#34 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (libkudu_test_main.so+0x7a1c)
Thread T20 'client-negotiat' (tid=14150, finished) created by thread T23 at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (security-itest+0x337755)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43719a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3939)
#3 kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:847:10 (libkudu_util.so+0x45442b)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:642:21 (libkudu_util.so+0x45282f)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:524:10 (libkudu_util.so+0x454992)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198df8)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x199851)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199416)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4dae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x198782)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5fd3)
#12 ev_invoke_pending /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9982)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x197655)
#14 ev_run /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab2c)
#15 ev::loop_ref::run(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a2768)
#16 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19ac35)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d981)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d939)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d8c9)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d891)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19cb8d)
#22 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x5fda4)
#23 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x5fbd9)
#24 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x437936)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126 in operator delete(void*)
==================
ThreadSanitizer: reported 3 warnings